Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:35.657183 8467 test_util.cc:268] Using random seed: 951748018
I20240502 08:16:35.669407 8467 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:35.669497 8467 ts_itest-base.cc:113] --------------
I20240502 08:16:35.669538 8467 ts_itest-base.cc:114] 8 tablet servers
I20240502 08:16:35.669585 8467 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:35.669632 8467 ts_itest-base.cc:116] --------------
2024-05-02T08:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:35Z Disabled control of system clock
I20240502 08:16:35.695297 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:40717
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:40717
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:16:36.053843 8479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.054216 8479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.054352 8479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.062234 8479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:36.062407 8479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:36.062505 8479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.062593 8479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:36.062666 8479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:36.073010 8479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:40717
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.074395 8479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.076280 8479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.089550 8485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.090039 8487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.090859 8484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.091820 8479 server_base.cc:1034] running on GCE node
I20240502 08:16:36.092653 8479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.094225 8479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.095438 8479 hybrid_clock.cc:648] HybridClock initialized: now 1714637796095373 us; error 59 us; skew 500 ppm
I20240502 08:16:36.095808 8479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.104257 8479 webserver.cc:460] Webserver started at http://127.8.68.254:45909/ using document root <none> and password file <none>
I20240502 08:16:36.105150 8479 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.105290 8479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.105679 8479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.108644 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "22efbb7ae91542d796b75fe914a8157b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.109663 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "22efbb7ae91542d796b75fe914a8157b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.116194 8479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 08:16:36.120213 8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.121891 8479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:36.122102 8479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "22efbb7ae91542d796b75fe914a8157b"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.122332 8479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.150130 8479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.151038 8479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.151376 8479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.172775 8479 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:40717
I20240502 08:16:36.172793 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:40717 every 8 connection(s)
I20240502 08:16:36.174518 8479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:36.179184 8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.184405 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 8479
I20240502 08:16:36.184724 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:36.191228 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b: Bootstrap starting.
I20240502 08:16:36.195708 8545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.197131 8545 log.cc:824] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:36.200644 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b: No bootstrap required, opened a new log
I20240502 08:16:36.211339 8545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } }
I20240502 08:16:36.211828 8545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.211952 8545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22efbb7ae91542d796b75fe914a8157b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.212774 8545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } }
I20240502 08:16:36.213054 8545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:36.213219 8545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:36.213445 8545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.216506 8545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } }
I20240502 08:16:36.217201 8545 leader_election.cc:302] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22efbb7ae91542d796b75fe914a8157b; no voters:
I20240502 08:16:36.217984 8545 leader_election.cc:288] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:36.218091 8550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:36.220286 8550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [term 1 LEADER]: Becoming Leader. State: Replica: 22efbb7ae91542d796b75fe914a8157b, State: Running, Role: LEADER
I20240502 08:16:36.220943 8550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } }
I20240502 08:16:36.221354 8545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:36.224347 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22efbb7ae91542d796b75fe914a8157b. Latest consensus state: current_term: 1 leader_uuid: "22efbb7ae91542d796b75fe914a8157b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } } }
I20240502 08:16:36.224836 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:36.225217 8551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22efbb7ae91542d796b75fe914a8157b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22efbb7ae91542d796b75fe914a8157b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 40717 } } }
I20240502 08:16:36.225499 8551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:36.232081 8556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:36.239209 8556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:36.249099 8556 catalog_manager.cc:1347] Generated new cluster ID: cad500ec99764c2b8ffc84d3fabe8673
I20240502 08:16:36.249244 8556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:36.263365 8556 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:36.264820 8556 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:36.278935 8556 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22efbb7ae91542d796b75fe914a8157b: Generated new TSK 0
I20240502 08:16:36.280164 8556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:36.292083 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 08:16:36.662933 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.663367 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.663589 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.673796 8569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:36.673981 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.674189 8569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240502 08:16:36.686579 8569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.688167 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.690328 8569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.706741 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.708536 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.708715 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.709905 8569 server_base.cc:1034] running on GCE node
I20240502 08:16:36.710676 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.712034 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.713289 8569 hybrid_clock.cc:648] HybridClock initialized: now 1714637796713202 us; error 52 us; skew 500 ppm
I20240502 08:16:36.713691 8569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.716776 8569 webserver.cc:460] Webserver started at http://127.8.68.193:46605/ using document root <none> and password file <none>
I20240502 08:16:36.717712 8569 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.717916 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.718344 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.723141 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bdf1a2f21da04697b58f19f331492b84"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.724529 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bdf1a2f21da04697b58f19f331492b84"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.732452 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20240502 08:16:36.736335 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.737831 8569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 08:16:36.737998 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bdf1a2f21da04697b58f19f331492b84"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:16:36.738224 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.759495 8569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.760401 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.760715 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.761710 8569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.763756 8569 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.763849 8569 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.763976 8569 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.764056 8569 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.812893 8569 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:34415
I20240502 08:16:36.812942 8695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:34415 every 8 connection(s)
I20240502 08:16:36.814308 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:36.822072 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 8569
I20240502 08:16:36.822302 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:36.825980 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:36.828575 8696 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:36.828878 8696 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.829764 8696 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:36.831905 8510 ts_manager.cc:194] Registered new tserver with Master: bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:36.833420 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:46461
W20240502 08:16:37.290954 8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:37.291309 8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:37.291431 8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:37.299109 8700 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:37.299316 8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:37.299470 8700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240502 08:16:37.309548 8700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:37.310938 8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:37.312798 8700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:37.326759 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.327503 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.328285 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:37.329152 8700 server_base.cc:1034] running on GCE node
I20240502 08:16:37.329874 8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:37.330914 8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:37.332134 8700 hybrid_clock.cc:648] HybridClock initialized: now 1714637797332053 us; error 71 us; skew 500 ppm
I20240502 08:16:37.332468 8700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:37.335124 8700 webserver.cc:460] Webserver started at http://127.8.68.194:40965/ using document root <none> and password file <none>
I20240502 08:16:37.335976 8700 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:37.336100 8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:37.336511 8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:37.339438 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "14ea87db2f6a4497a2a961afecd23a32"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.340281 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "14ea87db2f6a4497a2a961afecd23a32"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.346058 8700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:37.350517 8714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.352149 8700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:37.352334 8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14ea87db2f6a4497a2a961afecd23a32"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.352556 8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:37.376330 8700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:37.377254 8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:37.377539 8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:37.378649 8700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:37.380803 8700 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:37.380918 8700 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.381040 8700 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:37.381117 8700 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.423722 8700 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:46207
I20240502 08:16:37.423779 8826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:46207 every 8 connection(s)
I20240502 08:16:37.425118 8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:37.433147 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 8700
I20240502 08:16:37.433341 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:37.436909 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:37.439826 8827 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:37.440158 8827 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:37.441005 8827 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:37.442610 8510 ts_manager.cc:194] Registered new tserver with Master: 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:16:37.443491 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:41303
W20240502 08:16:37.789760 8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:37.790086 8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:37.790189 8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:37.797845 8831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:37.797994 8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:37.798120 8831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240502 08:16:37.807776 8831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:37.809106 8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:37.810824 8831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:37.824250 8836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.825124 8837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.825723 8839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:37.826813 8831 server_base.cc:1034] running on GCE node
I20240502 08:16:37.827656 8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:37.828632 8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:37.829882 8831 hybrid_clock.cc:648] HybridClock initialized: now 1714637797829788 us; error 57 us; skew 500 ppm
I20240502 08:16:37.830292 8831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:37.832916 8831 webserver.cc:460] Webserver started at http://127.8.68.195:34481/ using document root <none> and password file <none>
I20240502 08:16:37.833685 8831 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:37.833796 8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:37.834136 8831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:37.836274 8696 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:37.837261 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "71df2128b229478facd8ba1b2683ccca"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.838186 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "71df2128b229478facd8ba1b2683ccca"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.844098 8831 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 08:16:37.847563 8845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.848951 8831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 08:16:37.849118 8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "71df2128b229478facd8ba1b2683ccca"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-2dhs"
I20240502 08:16:37.849316 8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:37.863469 8831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:37.864331 8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:37.864619 8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:37.865618 8831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:37.867748 8831 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:37.867849 8831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.867964 8831 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:37.868037 8831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.904418 8831 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:36747
I20240502 08:16:37.904479 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:36747 every 8 connection(s)
I20240502 08:16:37.905805 8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:37.907423 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 8831
I20240502 08:16:37.907620 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:37.910874 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:37.920086 8958 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:37.920469 8958 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:37.921625 8958 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:37.923481 8510 ts_manager.cc:194] Registered new tserver with Master: 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:37.924345 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:39645
W20240502 08:16:38.288444 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:38.288741 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:38.288880 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:38.296638 8961 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:38.296780 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:38.297041 8961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240502 08:16:38.307011 8961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:38.308422 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:38.310163 8961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.323544 8966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.324211 8969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.325052 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.326035 8961 server_base.cc:1034] running on GCE node
I20240502 08:16:38.326674 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:38.327767 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:38.329005 8961 hybrid_clock.cc:648] HybridClock initialized: now 1714637798328961 us; error 46 us; skew 500 ppm
I20240502 08:16:38.329342 8961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.331980 8961 webserver.cc:460] Webserver started at http://127.8.68.196:39577/ using document root <none> and password file <none>
I20240502 08:16:38.332803 8961 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.332950 8961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.333272 8961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.336023 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "78c8d87aa5e44674959cd90af335de6a"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.336838 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "78c8d87aa5e44674959cd90af335de6a"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.342911 8961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 08:16:38.346522 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.348096 8961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:38.348268 8961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "78c8d87aa5e44674959cd90af335de6a"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.348472 8961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.371564 8961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.372479 8961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:38.372782 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.373807 8961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:38.375387 8961 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:38.375484 8961 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.375607 8961 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:38.375679 8961 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.421701 8961 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:35907
I20240502 08:16:38.421841 9087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:35907 every 8 connection(s)
I20240502 08:16:38.423486 8961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:38.431929 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 8961
I20240502 08:16:38.432119 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:38.438244 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:38.441483 9088 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:38.441857 9088 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.443038 9088 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:38.444833 8510 ts_manager.cc:194] Registered new tserver with Master: 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:16:38.446141 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:50311
I20240502 08:16:38.446261 8827 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
W20240502 08:16:38.800015 9092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:38.800356 9092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:38.800523 9092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:38.808166 9092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:38.808331 9092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:38.808542 9092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:16:38.818957 9092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:38.820297 9092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:38.822170 9092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:38.835999 9098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.836459 9097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:38.837378 9100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:38.838239 9092 server_base.cc:1034] running on GCE node
I20240502 08:16:38.838945 9092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:38.840140 9092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:38.841358 9092 hybrid_clock.cc:648] HybridClock initialized: now 1714637798841300 us; error 56 us; skew 500 ppm
I20240502 08:16:38.841691 9092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:38.844146 9092 webserver.cc:460] Webserver started at http://127.8.68.197:39735/ using document root <none> and password file <none>
I20240502 08:16:38.844918 9092 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:38.845047 9092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:38.845362 9092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:38.848037 9092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d1f0504c982c42b8b9de29b8830c1e82"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.848800 9092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d1f0504c982c42b8b9de29b8830c1e82"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.854530 9092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 08:16:38.858714 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.860535 9092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 08:16:38.860690 9092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1f0504c982c42b8b9de29b8830c1e82"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:38.860872 9092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:38.881659 9092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:38.882508 9092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:38.882761 9092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:38.884284 9092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:38.885790 9092 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:38.885881 9092 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.885968 9092 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:38.886032 9092 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:38.927871 8958 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:38.929688 9092 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:37561
I20240502 08:16:38.929734 9218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:37561 every 8 connection(s)
I20240502 08:16:38.931073 9092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:38.941414 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 9092
I20240502 08:16:38.941704 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:38.945225 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:38.945499 9219 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:38.945791 9219 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.946656 9219 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:38.948105 8510 ts_manager.cc:194] Registered new tserver with Master: d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561)
I20240502 08:16:38.948923 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:38427
W20240502 08:16:39.306623 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:39.306955 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:39.307117 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:39.315537 9222 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:39.315703 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:39.315886 9222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:16:39.326134 9222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:39.327618 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:39.329447 9222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:39.343623 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.344408 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.344971 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:39.345880 9222 server_base.cc:1034] running on GCE node
I20240502 08:16:39.346531 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:39.347613 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:39.348836 9222 hybrid_clock.cc:648] HybridClock initialized: now 1714637799348788 us; error 53 us; skew 500 ppm
I20240502 08:16:39.349190 9222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:39.352424 9222 webserver.cc:460] Webserver started at http://127.8.68.198:40561/ using document root <none> and password file <none>
I20240502 08:16:39.353330 9222 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:39.353461 9222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:39.353796 9222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:39.356767 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4586652e013946998246ca5489ae7083"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.358346 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4586652e013946998246ca5489ae7083"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.364064 9222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:16:39.367998 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.369789 9222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:39.370002 9222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4586652e013946998246ca5489ae7083"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.370232 9222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:39.396072 9222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:39.397078 9222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:39.397380 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:39.398535 9222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:39.400177 9222 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:39.400283 9222 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.400408 9222 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:39.400487 9222 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.443296 9222 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:36779
I20240502 08:16:39.443456 9348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:36779 every 8 connection(s)
I20240502 08:16:39.444833 9222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:39.446051 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 9222
I20240502 08:16:39.446252 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:16:39.450382 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:0
--local_ip_for_outbound_sockets=127.8.68.199
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:39.452179 9088 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:39.460474 9349 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:39.460839 9349 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:39.461933 9349 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:39.463627 8510 ts_manager.cc:194] Registered new tserver with Master: 4586652e013946998246ca5489ae7083 (127.8.68.198:36779)
I20240502 08:16:39.464473 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:53609
W20240502 08:16:39.811101 9353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:39.811475 9353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:39.812129 9353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:39.821484 9353 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:39.821616 9353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:39.821792 9353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:16:39.831696 9353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:39.833096 9353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:39.835068 9353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:39.848095 9358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.848973 9361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.849630 9359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:39.850759 9353 server_base.cc:1034] running on GCE node
I20240502 08:16:39.851526 9353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:39.852603 9353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:39.853811 9353 hybrid_clock.cc:648] HybridClock initialized: now 1714637799853755 us; error 46 us; skew 500 ppm
I20240502 08:16:39.854158 9353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:39.856750 9353 webserver.cc:460] Webserver started at http://127.8.68.199:42939/ using document root <none> and password file <none>
I20240502 08:16:39.857556 9353 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:39.857669 9353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:39.858007 9353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:39.860760 9353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "eae55bf90c524e8eb9df4c31f40096e4"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.861562 9353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "eae55bf90c524e8eb9df4c31f40096e4"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.867306 9353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:16:39.871027 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.872491 9353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:39.872678 9353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eae55bf90c524e8eb9df4c31f40096e4"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:39.872879 9353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:39.895915 9353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:39.896808 9353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:39.897081 9353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:39.898165 9353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:39.899713 9353 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:39.899804 9353 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.899924 9353 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:39.899998 9353 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.944094 9353 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:45071
I20240502 08:16:39.944142 9479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:45071 every 8 connection(s)
I20240502 08:16:39.945515 9353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:16:39.951437 9219 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:39.952517 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 9353
I20240502 08:16:39.952728 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:16:39.956784 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:0
--local_ip_for_outbound_sockets=127.8.68.200
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:40717
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:39.961143 9480 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:39.961467 9480 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:39.962620 9480 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:39.964241 8510 ts_manager.cc:194] Registered new tserver with Master: eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071)
I20240502 08:16:39.965466 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:47127
W20240502 08:16:40.352130 9483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:40.352443 9483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:40.352592 9483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:40.361423 9483 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:40.361557 9483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:40.361845 9483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240502 08:16:40.372087 9483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:40.373862 9483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:40.375823 9483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:40.392068 9488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.392812 9489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.393450 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:40.394250 9483 server_base.cc:1034] running on GCE node
I20240502 08:16:40.395153 9483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:40.396631 9483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:40.397920 9483 hybrid_clock.cc:648] HybridClock initialized: now 1714637800397836 us; error 63 us; skew 500 ppm
I20240502 08:16:40.398411 9483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:40.402575 9483 webserver.cc:460] Webserver started at http://127.8.68.200:45707/ using document root <none> and password file <none>
I20240502 08:16:40.403746 9483 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:40.403915 9483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:40.404371 9483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:40.408810 9483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-2dhs"
I20240502 08:16:40.410128 9483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-2dhs"
I20240502 08:16:40.418498 9483 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20240502 08:16:40.424166 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.426190 9483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:40.426443 9483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-2dhs"
I20240502 08:16:40.426924 9483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:40.468005 9349 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:40.470767 9483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:40.471990 9483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:40.472343 9483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:40.473371 9483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:40.475102 9483 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:40.475206 9483 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.475358 9483 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:40.475489 9483 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.521042 9483 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:32925
I20240502 08:16:40.521189 9609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:32925 every 8 connection(s)
I20240502 08:16:40.522954 9483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:16:40.530582 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 9483
I20240502 08:16:40.530758 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:16:40.537756 9610 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:40.538096 9610 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:40.538951 9610 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:40.540386 8510 ts_manager.cc:194] Registered new tserver with Master: 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925)
I20240502 08:16:40.541154 8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:54171
I20240502 08:16:40.547550 8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:16:40.570417 8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:16:40.570595 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 9092
I20240502 08:16:40.579466 9214 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:40.659626 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 9092
I20240502 08:16:40.676578 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 9222
I20240502 08:16:40.685607 9344 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:40.766189 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 9222
I20240502 08:16:40.790141 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 9353
I20240502 08:16:40.798336 9475 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:40.881454 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 9353
I20240502 08:16:40.898394 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 9483
I20240502 08:16:40.906855 9605 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:40.990182 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 9483
I20240502 08:16:44.760627 8509 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56274:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:44.808676 8631 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8f8d5081284a3288dab20e0e19d9bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.809326 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 888fe8d0871a4a539b43569e0eda4bef (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.808102 8630 tablet_service.cc:1467] Processing CreateTablet for tablet 295aa577b3bf4bfebf8870b8a78aaa12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.810101 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.810591 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8f8d5081284a3288dab20e0e19d9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.811270 9023 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8f8d5081284a3288dab20e0e19d9bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.812009 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295aa577b3bf4bfebf8870b8a78aaa12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.812654 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8f8d5081284a3288dab20e0e19d9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.815762 8762 tablet_service.cc:1467] Processing CreateTablet for tablet 295aa577b3bf4bfebf8870b8a78aaa12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.816306 8892 tablet_service.cc:1467] Processing CreateTablet for tablet 295aa577b3bf4bfebf8870b8a78aaa12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.817262 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295aa577b3bf4bfebf8870b8a78aaa12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.817656 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295aa577b3bf4bfebf8870b8a78aaa12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.821179 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8f8d5081284a3288dab20e0e19d9bf (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.821660 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8f8d5081284a3288dab20e0e19d9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.825835 8761 tablet_service.cc:1467] Processing CreateTablet for tablet 888fe8d0871a4a539b43569e0eda4bef (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.826429 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.811036 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 888fe8d0871a4a539b43569e0eda4bef (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.827497 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.832402 9640 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:44.836589 9640 tablet_bootstrap.cc:654] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.838064 9640 log.cc:824] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.847930 9640 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:44.848316 9640 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20240502 08:16:44.865799 9643 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:44.869220 9643 tablet_bootstrap.cc:654] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.869684 9644 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:44.870656 9643 log.cc:824] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.871743 9640 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.872262 9640 raft_consensus.cc:374] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.872260 9642 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:44.872383 9640 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.873289 9640 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.874184 9644 tablet_bootstrap.cc:654] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.874338 9643 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:44.874631 9643 ts_tablet_manager.cc:1367] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:16:44.875854 9642 tablet_bootstrap.cc:654] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.876147 9644 log.cc:824] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.877096 9642 log.cc:824] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.882145 9640 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.034s user 0.015s sys 0.006s
I20240502 08:16:44.887840 9643 raft_consensus.cc:348] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.888535 9643 raft_consensus.cc:374] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.888679 9643 raft_consensus.cc:729] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.889844 9640 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:44.890806 9643 consensus_queue.cc:260] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.891528 9640 tablet_bootstrap.cc:654] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.892225 9642 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:44.892386 9644 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:44.892613 9642 ts_tablet_manager.cc:1367] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20240502 08:16:44.892860 9644 ts_tablet_manager.cc:1367] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.012s
I20240502 08:16:44.905419 9643 ts_tablet_manager.cc:1398] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.031s user 0.025s sys 0.005s
I20240502 08:16:44.905896 9643 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:44.905803 9642 raft_consensus.cc:348] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.906288 9642 raft_consensus.cc:374] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.906406 9642 raft_consensus.cc:729] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.907338 9642 consensus_queue.cc:260] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.908249 9643 tablet_bootstrap.cc:654] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.913167 9644 raft_consensus.cc:348] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.913759 9644 raft_consensus.cc:374] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.913892 9644 raft_consensus.cc:729] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
W20240502 08:16:44.913949 8959 tablet.cc:2369] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:44.914861 9644 consensus_queue.cc:260] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.920671 9640 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:44.920905 9640 ts_tablet_manager.cc:1367] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.011s
I20240502 08:16:44.922492 9640 raft_consensus.cc:348] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.922804 9640 raft_consensus.cc:374] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.922894 9640 raft_consensus.cc:729] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.923302 9640 consensus_queue.cc:260] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.924058 9640 ts_tablet_manager.cc:1398] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:44.925279 9640 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:44.933466 9640 tablet_bootstrap.cc:654] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.937824 9643 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:44.937978 9642 ts_tablet_manager.cc:1398] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.045s user 0.028s sys 0.017s
I20240502 08:16:44.938059 9643 ts_tablet_manager.cc:1367] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.007s
I20240502 08:16:44.938436 9642 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:44.939486 9643 raft_consensus.cc:348] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.939781 9643 raft_consensus.cc:374] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.939867 9643 raft_consensus.cc:729] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.940156 9642 tablet_bootstrap.cc:654] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.940181 9643 consensus_queue.cc:260] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.940804 9643 ts_tablet_manager.cc:1398] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:44.943605 9644 ts_tablet_manager.cc:1398] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.051s user 0.015s sys 0.024s
I20240502 08:16:44.944131 9644 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:44.946291 9644 tablet_bootstrap.cc:654] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.946405 9640 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:44.946645 9640 ts_tablet_manager.cc:1367] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 08:16:44.947742 9640 raft_consensus.cc:348] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.949519 9640 raft_consensus.cc:374] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.949625 9640 raft_consensus.cc:729] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.949975 9640 consensus_queue.cc:260] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.951364 9640 ts_tablet_manager.cc:1398] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:44.953087 9642 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:44.953096 9644 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:44.953255 9644 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 08:16:44.953334 9642 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240502 08:16:44.954418 9644 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.954421 9642 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.954646 9642 raft_consensus.cc:374] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.954646 9644 raft_consensus.cc:374] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.954743 9642 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.954751 9644 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.954974 9642 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.954977 9644 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:44.955549 9642 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:44.955605 9644 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:44.963579 9651 raft_consensus.cc:482] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.963819 9651 raft_consensus.cc:504] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.965734 9651 leader_election.cc:288] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:44.973656 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8f8d5081284a3288dab20e0e19d9bf" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:44.973954 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8f8d5081284a3288dab20e0e19d9bf" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:44.974185 8651 raft_consensus.cc:2454] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:44.974413 8913 raft_consensus.cc:2454] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:44.974856 8978 leader_election.cc:302] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c8d87aa5e44674959cd90af335de6a, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:44.975299 9651 raft_consensus.cc:2781] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.975435 9651 raft_consensus.cc:482] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.975531 9651 raft_consensus.cc:3037] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.977579 9651 raft_consensus.cc:504] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.978233 9651 leader_election.cc:288] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Requested vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:44.978854 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8f8d5081284a3288dab20e0e19d9bf" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:44.978854 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8f8d5081284a3288dab20e0e19d9bf" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:44.979081 8913 raft_consensus.cc:3037] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.979080 8651 raft_consensus.cc:3037] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.982280 8913 raft_consensus.cc:2454] T 3b8f8d5081284a3288dab20e0e19d9bf P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:44.982277 8651 raft_consensus.cc:2454] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:44.982808 8976 leader_election.cc:302] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:44.983137 9651 raft_consensus.cc:2781] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.983587 9651 raft_consensus.cc:686] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:16:44.984062 9651 consensus_queue.cc:237] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.988215 9649 raft_consensus.cc:482] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.988411 9649 raft_consensus.cc:504] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:44.989217 8510 catalog_manager.cc:5484] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:44.989949 9649 leader_election.cc:288] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:44.997404 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295aa577b3bf4bfebf8870b8a78aaa12" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:44.997929 8782 raft_consensus.cc:2454] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 0.
I20240502 08:16:44.997938 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295aa577b3bf4bfebf8870b8a78aaa12" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:44.998162 8651 raft_consensus.cc:2454] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 0.
I20240502 08:16:44.998502 8847 leader_election.cc:302] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters:
I20240502 08:16:44.998899 9649 raft_consensus.cc:2781] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.999080 9649 raft_consensus.cc:482] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.999189 9649 raft_consensus.cc:3037] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.001168 9649 raft_consensus.cc:504] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.001736 9649 leader_election.cc:288] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:45.002202 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295aa577b3bf4bfebf8870b8a78aaa12" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:45.002254 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295aa577b3bf4bfebf8870b8a78aaa12" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:45.002420 8782 raft_consensus.cc:3037] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.002482 8651 raft_consensus.cc:3037] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.004817 8651 raft_consensus.cc:2454] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:45.004880 8782 raft_consensus.cc:2454] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:45.005250 8848 leader_election.cc:302] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:45.005599 9649 raft_consensus.cc:2781] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:45.006021 9649 raft_consensus.cc:686] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Becoming Leader. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:16:45.006714 9649 consensus_queue.cc:237] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.011293 8510 catalog_manager.cc:5484] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca reported cstate change: term changed from 0 to 1, leader changed from <none> to 71df2128b229478facd8ba1b2683ccca (127.8.68.195). New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:45.058822 9653 raft_consensus.cc:482] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.059044 9653 raft_consensus.cc:504] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.060830 9653 leader_election.cc:288] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:45.068662 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:45.068923 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:45.069056 9043 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:45.069155 8651 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:45.069638 8715 leader_election.cc:302] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:45.070114 9653 raft_consensus.cc:2781] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:45.070231 9653 raft_consensus.cc:482] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:45.070324 9653 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:45.072479 8697 tablet.cc:2369] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:45.072423 9653 raft_consensus.cc:504] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.073112 9653 leader_election.cc:288] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:45.073621 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:45.073627 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:45.073787 8651 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.073799 9043 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.075820 8651 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:45.075894 9043 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:45.076364 8717 leader_election.cc:302] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:45.076687 9653 raft_consensus.cc:2781] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:45.077085 9653 raft_consensus.cc:686] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:45.077629 9653 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.082374 8510 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
W20240502 08:16:45.181813 9089 tablet.cc:2369] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:45.182708 8828 tablet.cc:2369] T 295aa577b3bf4bfebf8870b8a78aaa12 P 14ea87db2f6a4497a2a961afecd23a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:45.381527 9651 consensus_queue.cc:1035] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.390039 9651 consensus_queue.cc:1035] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:45.468051 9669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:45.468416 9669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:45.476109 9669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:45.480224 9649 consensus_queue.cc:1035] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.485751 9649 consensus_queue.cc:1035] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.514933 9653 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 08:16:45.521147 9690 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 277.206 ms
time per row: 0.138603 ms
I20240502 08:16:45.925681 8503 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56274:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:45.936043 9023 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f85e56bc14589b9535fd1efe64e9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.936529 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f85e56bc14589b9535fd1efe64e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.937307 8892 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f85e56bc14589b9535fd1efe64e9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.937747 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f85e56bc14589b9535fd1efe64e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.937796 8631 tablet_service.cc:1467] Processing CreateTablet for tablet 49af926b61e64eb0ac9c4dcd5b638152 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.938256 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49af926b61e64eb0ac9c4dcd5b638152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.940220 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2de1aca9a44a489972aab5decf3620 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.940649 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2de1aca9a44a489972aab5decf3620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.941871 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2de1aca9a44a489972aab5decf3620 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.942277 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2de1aca9a44a489972aab5decf3620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.942813 8762 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f85e56bc14589b9535fd1efe64e9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.942956 8760 tablet_service.cc:1467] Processing CreateTablet for tablet 49af926b61e64eb0ac9c4dcd5b638152 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.943287 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f85e56bc14589b9535fd1efe64e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.943595 8761 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2de1aca9a44a489972aab5decf3620 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.943830 8760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49af926b61e64eb0ac9c4dcd5b638152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.944168 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2de1aca9a44a489972aab5decf3620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.953500 9738 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:45.955614 9738 tablet_bootstrap.cc:654] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.957773 8891 tablet_service.cc:1467] Processing CreateTablet for tablet 49af926b61e64eb0ac9c4dcd5b638152 (DEFAULT_TABLE table=rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.958571 8891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49af926b61e64eb0ac9c4dcd5b638152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.958961 9737 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:45.960628 9737 tablet_bootstrap.cc:654] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.961380 9738 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:45.961637 9738 ts_tablet_manager.cc:1367] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:16:45.962599 9737 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:45.962518 9738 raft_consensus.cc:348] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.962711 9738 raft_consensus.cc:374] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.962755 9737 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:45.962805 9738 raft_consensus.cc:729] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.963158 9738 consensus_queue.cc:260] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.963557 9737 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.963760 9737 raft_consensus.cc:374] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.963829 9738 ts_tablet_manager.cc:1398] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:45.963840 9737 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.964103 9737 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.964728 9737 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:45.971904 9742 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:45.973969 9742 tablet_bootstrap.cc:654] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.978196 9738 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:45.978881 9739 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:45.979936 9738 tablet_bootstrap.cc:654] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.980089 9742 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:45.980247 9742 ts_tablet_manager.cc:1367] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:16:45.980543 9739 tablet_bootstrap.cc:654] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.981297 9742 raft_consensus.cc:348] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.981506 9742 raft_consensus.cc:374] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.981632 9742 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.982040 9738 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:45.981891 9742 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.982204 9738 ts_tablet_manager.cc:1367] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:45.982470 9742 ts_tablet_manager.cc:1398] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:45.982776 9742 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:45.982971 9738 raft_consensus.cc:348] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.983179 9738 raft_consensus.cc:374] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.983276 9738 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.983541 9738 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.983990 9646 raft_consensus.cc:482] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.984087 9738 ts_tablet_manager.cc:1398] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:45.984182 9646 raft_consensus.cc:504] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:45.986289 9646 leader_election.cc:288] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:45.993729 9742 tablet_bootstrap.cc:654] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.994225 9739 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:45.994460 9739 ts_tablet_manager.cc:1367] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240502 08:16:45.995635 9739 raft_consensus.cc:348] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.995927 9739 raft_consensus.cc:374] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.996032 9739 raft_consensus.cc:729] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.996376 9739 consensus_queue.cc:260] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:45.997130 9739 ts_tablet_manager.cc:1398] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:45.997651 9739 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:45.999545 9742 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:45.999727 9742 ts_tablet_manager.cc:1367] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20240502 08:16:46.000161 9739 tablet_bootstrap.cc:654] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.000548 9742 raft_consensus.cc:348] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.000795 9742 raft_consensus.cc:374] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.000869 9742 raft_consensus.cc:729] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.001137 9742 consensus_queue.cc:260] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.001675 9742 ts_tablet_manager.cc:1398] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.002003 9742 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:46.004112 9739 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:46.004273 9739 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:16:46.004954 9739 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.005198 9739 raft_consensus.cc:374] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.005292 9739 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.005568 9739 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.006278 9739 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.007083 9739 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:46.008848 9739 tablet_bootstrap.cc:654] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.010983 9742 tablet_bootstrap.cc:654] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.013926 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
W20240502 08:16:46.014961 8584 leader_election.cc:341] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.015401 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:46.015709 8781 raft_consensus.cc:2454] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 0.
I20240502 08:16:46.016201 8585 leader_election.cc:302] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters: 71df2128b229478facd8ba1b2683ccca
I20240502 08:16:46.016613 9646 raft_consensus.cc:2781] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.016747 9646 raft_consensus.cc:482] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.016844 9646 raft_consensus.cc:3037] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.017103 9739 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:46.017258 9739 ts_tablet_manager.cc:1367] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240502 08:16:46.017573 9742 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:46.017715 9742 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240502 08:16:46.018090 9739 raft_consensus.cc:348] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.018294 9739 raft_consensus.cc:374] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.018381 9739 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.018342 9742 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.018539 9742 raft_consensus.cc:374] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.018636 9742 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.018616 9739 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.018867 9742 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.019176 9739 ts_tablet_manager.cc:1398] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.019451 9742 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.019745 9646 raft_consensus.cc:504] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.020961 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:46.021162 8781 raft_consensus.cc:3037] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.021292 9646 leader_election.cc:288] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:46.021837 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:46.022043 8913 raft_consensus.cc:3037] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.023780 8781 raft_consensus.cc:2454] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 1.
I20240502 08:16:46.024168 8585 leader_election.cc:302] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:46.024271 8913 raft_consensus.cc:2454] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 1.
I20240502 08:16:46.024485 9646 raft_consensus.cc:2781] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.025166 9646 raft_consensus.cc:686] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 1 LEADER]: Becoming Leader. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Running, Role: LEADER
I20240502 08:16:46.025702 9646 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:46.029352 8503 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf1a2f21da04697b58f19f331492b84 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "bdf1a2f21da04697b58f19f331492b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:46.060786 9690 raft_consensus.cc:482] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.061007 9690 raft_consensus.cc:504] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.061789 9690 leader_election.cc:288] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:46.062394 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f85e56bc14589b9535fd1efe64e9d" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:46.062640 9043 raft_consensus.cc:2454] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:46.063058 8715 leader_election.cc:302] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:46.063385 9690 raft_consensus.cc:2781] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.063513 9690 raft_consensus.cc:482] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.063724 9690 raft_consensus.cc:3037] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.066428 9690 raft_consensus.cc:504] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.067039 9690 leader_election.cc:288] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:46.067600 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f85e56bc14589b9535fd1efe64e9d" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:46.067840 9043 raft_consensus.cc:3037] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.069648 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f85e56bc14589b9535fd1efe64e9d" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:46.069737 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f85e56bc14589b9535fd1efe64e9d" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:46.069940 8913 raft_consensus.cc:2454] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:46.070173 9043 raft_consensus.cc:2454] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:46.070734 8715 leader_election.cc:302] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters: 71df2128b229478facd8ba1b2683ccca
I20240502 08:16:46.071115 9690 raft_consensus.cc:2781] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.071293 9690 raft_consensus.cc:686] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:46.071557 9690 consensus_queue.cc:237] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.074281 8503 catalog_manager.cc:5484] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:46.084273 9716 raft_consensus.cc:482] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.084496 9716 raft_consensus.cc:504] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.085285 9716 leader_election.cc:288] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:46.085980 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:46.086277 8912 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:46.086711 8976 leader_election.cc:302] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:46.086982 9716 raft_consensus.cc:2781] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.087148 9716 raft_consensus.cc:482] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.087280 9716 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.089267 9716 raft_consensus.cc:504] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.089916 9716 leader_election.cc:288] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:46.090448 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:46.090691 8912 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.092309 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:46.092438 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:46.092551 8781 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:46.092947 8912 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:46.093338 8976 leader_election.cc:302] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters: 14ea87db2f6a4497a2a961afecd23a32
I20240502 08:16:46.093659 9716 raft_consensus.cc:2781] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.093778 9716 raft_consensus.cc:686] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:16:46.094084 9716 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:46.096879 8503 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:46.439903 9646 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.458709 9646 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:46.485081 9758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:46.485407 9758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:46.492964 9758 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:46.509466 8913 raft_consensus.cc:3037] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.512841 9690 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.513962 8782 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.518210 9653 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.519861 9716 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.525729 9716 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 258.54 ms
time per row: 0.12927 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:16:47.887984 8504 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56274:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:47.897444 8631 tablet_service.cc:1467] Processing CreateTablet for tablet 25df2da40e524e9c95cbe13a484445b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.897430 8762 tablet_service.cc:1467] Processing CreateTablet for tablet 25df2da40e524e9c95cbe13a484445b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.897513 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 25df2da40e524e9c95cbe13a484445b5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.897940 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.897958 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.897975 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.898973 8630 tablet_service.cc:1467] Processing CreateTablet for tablet 171c663b69ed42f9883aca8e7392cded (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.899425 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171c663b69ed42f9883aca8e7392cded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.903749 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 5553c68dc6b24e5bbd849453a0692a73 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.904264 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553c68dc6b24e5bbd849453a0692a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.904337 8891 tablet_service.cc:1467] Processing CreateTablet for tablet 171c663b69ed42f9883aca8e7392cded (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.904783 8891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171c663b69ed42f9883aca8e7392cded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.904774 9023 tablet_service.cc:1467] Processing CreateTablet for tablet 171c663b69ed42f9883aca8e7392cded (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.905150 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171c663b69ed42f9883aca8e7392cded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.908059 9021 tablet_service.cc:1467] Processing CreateTablet for tablet 5553c68dc6b24e5bbd849453a0692a73 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.908608 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553c68dc6b24e5bbd849453a0692a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.912544 8761 tablet_service.cc:1467] Processing CreateTablet for tablet 5553c68dc6b24e5bbd849453a0692a73 (DEFAULT_TABLE table=rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.913172 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5553c68dc6b24e5bbd849453a0692a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.920454 9829 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:47.924683 9827 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:47.926429 9827 tablet_bootstrap.cc:654] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.928038 9829 tablet_bootstrap.cc:654] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.930153 9827 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:47.930399 9827 ts_tablet_manager.cc:1367] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:16:47.931218 9827 raft_consensus.cc:348] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.931564 9827 raft_consensus.cc:374] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.931697 9827 raft_consensus.cc:729] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.932113 9828 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:47.932032 9827 consensus_queue.cc:260] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.934732 9828 tablet_bootstrap.cc:654] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.936725 9832 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:47.938549 9829 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:47.938755 9829 ts_tablet_manager.cc:1367] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240502 08:16:47.939229 9832 tablet_bootstrap.cc:654] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.939389 9827 ts_tablet_manager.cc:1398] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:16:47.942781 9827 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:47.942981 9829 raft_consensus.cc:348] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.943400 9829 raft_consensus.cc:374] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.943528 9829 raft_consensus.cc:729] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.943869 9829 consensus_queue.cc:260] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.946578 9827 tablet_bootstrap.cc:654] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.947664 9829 ts_tablet_manager.cc:1398] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:16:47.948037 9829 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:47.948366 9832 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:47.948596 9832 ts_tablet_manager.cc:1367] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 08:16:47.949566 9832 raft_consensus.cc:348] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.949904 9832 raft_consensus.cc:374] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.950171 9832 raft_consensus.cc:729] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.950552 9832 consensus_queue.cc:260] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.951050 9829 tablet_bootstrap.cc:654] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.951484 9832 ts_tablet_manager.cc:1398] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:47.953141 9832 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:47.955441 9827 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:47.955703 9827 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20240502 08:16:47.956164 9832 tablet_bootstrap.cc:654] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.956744 9827 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.957082 9827 raft_consensus.cc:374] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.957226 9827 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.957485 9827 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.958160 9827 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.958593 9829 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:47.958752 9829 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240502 08:16:47.958786 9832 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:47.958925 9832 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:47.958961 9828 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:47.959123 9828 ts_tablet_manager.cc:1367] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.008s
I20240502 08:16:47.959573 9829 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.959642 9832 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.959801 9829 raft_consensus.cc:374] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.959841 9832 raft_consensus.cc:374] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.959899 9829 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.959909 9832 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.959990 9828 raft_consensus.cc:348] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.960211 9828 raft_consensus.cc:374] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.960183 9832 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.960183 9829 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:47.960332 9828 raft_consensus.cc:729] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.960572 9828 consensus_queue.cc:260] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:47.961278 9832 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.961272 9829 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.961568 9829 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:47.961593 9827 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:47.962613 9828 ts_tablet_manager.cc:1398] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:47.963300 9827 tablet_bootstrap.cc:654] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.963292 9829 tablet_bootstrap.cc:654] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.970732 9829 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:47.970734 9827 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:47.970873 9829 ts_tablet_manager.cc:1367] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 08:16:47.970906 9827 ts_tablet_manager.cc:1367] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 08:16:47.971550 9829 raft_consensus.cc:348] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.971555 9827 raft_consensus.cc:348] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.971712 9829 raft_consensus.cc:374] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.971720 9827 raft_consensus.cc:374] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.971779 9829 raft_consensus.cc:729] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.971834 9827 raft_consensus.cc:729] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.972000 9829 consensus_queue.cc:260] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.972092 9827 consensus_queue.cc:260] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:47.972609 9827 ts_tablet_manager.cc:1398] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.972999 9829 ts_tablet_manager.cc:1398] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:48.025521 9651 raft_consensus.cc:482] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.025756 9651 raft_consensus.cc:504] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:48.026453 9651 leader_election.cc:288] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:48.027011 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171c663b69ed42f9883aca8e7392cded" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:48.027043 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171c663b69ed42f9883aca8e7392cded" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:48.027276 8650 raft_consensus.cc:2454] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:48.027326 8912 raft_consensus.cc:2454] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:48.027734 8978 leader_election.cc:302] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c8d87aa5e44674959cd90af335de6a, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:48.028030 9651 raft_consensus.cc:2781] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.028154 9651 raft_consensus.cc:482] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.028237 9651 raft_consensus.cc:3037] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.030093 9651 raft_consensus.cc:504] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:48.030992 9651 leader_election.cc:288] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Requested vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:48.031169 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171c663b69ed42f9883aca8e7392cded" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:48.031433 8650 raft_consensus.cc:3037] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.031416 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171c663b69ed42f9883aca8e7392cded" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:48.031607 8912 raft_consensus.cc:3037] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.034217 8912 raft_consensus.cc:2454] T 171c663b69ed42f9883aca8e7392cded P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:48.034540 8650 raft_consensus.cc:2454] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:48.034976 8976 leader_election.cc:302] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:48.035502 9651 raft_consensus.cc:2781] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.035979 9651 raft_consensus.cc:686] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:16:48.036311 9651 consensus_queue.cc:237] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:48.039233 8501 catalog_manager.cc:5484] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:48.099498 9653 raft_consensus.cc:482] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.099721 9653 raft_consensus.cc:504] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:48.100265 9653 leader_election.cc:288] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:48.100759 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5553c68dc6b24e5bbd849453a0692a73" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:48.100808 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5553c68dc6b24e5bbd849453a0692a73" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:48.101001 8650 raft_consensus.cc:2454] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:48.101047 9043 raft_consensus.cc:2454] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:48.101428 8717 leader_election.cc:302] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:48.101748 9653 raft_consensus.cc:2781] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.101847 9653 raft_consensus.cc:482] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.101965 9653 raft_consensus.cc:3037] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.104074 9653 raft_consensus.cc:504] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:48.104653 9653 leader_election.cc:288] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:48.105175 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5553c68dc6b24e5bbd849453a0692a73" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:48.105201 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5553c68dc6b24e5bbd849453a0692a73" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:48.105393 9043 raft_consensus.cc:3037] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.105422 8650 raft_consensus.cc:3037] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.108233 8650 raft_consensus.cc:2454] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:48.108246 9043 raft_consensus.cc:2454] T 5553c68dc6b24e5bbd849453a0692a73 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:48.108721 8717 leader_election.cc:302] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:48.109038 9653 raft_consensus.cc:2781] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.109416 9653 raft_consensus.cc:686] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:48.109695 9653 consensus_queue.cc:237] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:48.112234 8501 catalog_manager.cc:5484] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:48.120004 9651 raft_consensus.cc:482] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.120182 9651 raft_consensus.cc:504] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:48.120879 9651 leader_election.cc:288] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:16:48.121368 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:48.121495 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:48.121618 8650 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:48.121767 8781 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 0.
I20240502 08:16:48.122033 8978 leader_election.cc:302] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c8d87aa5e44674959cd90af335de6a, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:48.122298 9651 raft_consensus.cc:2781] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.122414 9651 raft_consensus.cc:482] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.122517 9651 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.124395 9651 raft_consensus.cc:504] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:48.125108 9651 leader_election.cc:288] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Requested vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:16:48.125548 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:48.125609 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:48.125732 8650 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.125825 8781 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.127780 8781 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:48.127939 8650 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 1.
I20240502 08:16:48.128260 8977 leader_election.cc:302] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:48.128553 9651 raft_consensus.cc:2781] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.128697 9651 raft_consensus.cc:686] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:16:48.128990 9651 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:48.131778 8501 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:48.513703 9653 consensus_queue.cc:1035] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:16:48.515182 9651 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.522181 9842 consensus_queue.cc:1035] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.540714 9651 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:48.549427 9845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:48.549767 9845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:48.557742 9845 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:16:48.590108 9651 consensus_queue.cc:1035] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.595330 9840 consensus_queue.cc:1035] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.746 ms
time per row: 0.123873 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:16:49.939428 8501 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56274:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:49.948812 8761 tablet_service.cc:1467] Processing CreateTablet for tablet c9479bf73bb1400c83c247f3654bb79b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.949312 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9479bf73bb1400c83c247f3654bb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.949491 8891 tablet_service.cc:1467] Processing CreateTablet for tablet c9479bf73bb1400c83c247f3654bb79b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.949924 8891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9479bf73bb1400c83c247f3654bb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.951040 9021 tablet_service.cc:1467] Processing CreateTablet for tablet e29f136434ad4ff1b0c9d4ac4481e216 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.951143 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 41873909d8db44f89580fb209502ab30 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.951488 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29f136434ad4ff1b0c9d4ac4481e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.951839 8892 tablet_service.cc:1467] Processing CreateTablet for tablet e29f136434ad4ff1b0c9d4ac4481e216 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.952215 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29f136434ad4ff1b0c9d4ac4481e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.952780 8762 tablet_service.cc:1467] Processing CreateTablet for tablet e29f136434ad4ff1b0c9d4ac4481e216 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.953132 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29f136434ad4ff1b0c9d4ac4481e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.955534 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41873909d8db44f89580fb209502ab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.955641 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 41873909d8db44f89580fb209502ab30 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.956007 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41873909d8db44f89580fb209502ab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.968943 9913 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:49.971532 9913 tablet_bootstrap.cc:654] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.972350 9914 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:49.972390 8629 tablet_service.cc:1467] Processing CreateTablet for tablet c9479bf73bb1400c83c247f3654bb79b (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.972832 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9479bf73bb1400c83c247f3654bb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.973157 9912 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:49.974193 9913 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:49.974298 9914 tablet_bootstrap.cc:654] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.974326 9913 ts_tablet_manager.cc:1367] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:16:49.974723 9912 tablet_bootstrap.cc:654] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.975178 9913 raft_consensus.cc:348] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:49.975445 9913 raft_consensus.cc:374] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.975534 9913 raft_consensus.cc:729] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.975827 9913 consensus_queue.cc:260] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:49.978873 9913 ts_tablet_manager.cc:1398] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:49.979274 9913 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:49.982370 9913 tablet_bootstrap.cc:654] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.983026 9912 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:49.983182 9912 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 08:16:49.984200 9912 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:49.984681 9912 raft_consensus.cc:374] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.984635 8630 tablet_service.cc:1467] Processing CreateTablet for tablet 41873909d8db44f89580fb209502ab30 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.984822 9912 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.985205 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41873909d8db44f89580fb209502ab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.985170 9912 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:49.989610 9912 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:49.993400 9912 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:49.995785 9912 tablet_bootstrap.cc:654] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.999464 9920 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:50.001591 9920 tablet_bootstrap.cc:654] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.002043 9913 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:50.002045 9912 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:50.002231 9912 ts_tablet_manager.cc:1367] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 08:16:50.002241 9913 ts_tablet_manager.cc:1367] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.014s
I20240502 08:16:50.002758 9914 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:50.002964 9914 ts_tablet_manager.cc:1367] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.020s
I20240502 08:16:50.003047 9913 raft_consensus.cc:348] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.003082 9912 raft_consensus.cc:348] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.003322 9913 raft_consensus.cc:374] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.003335 9912 raft_consensus.cc:374] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.003409 9913 raft_consensus.cc:729] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.003561 9912 raft_consensus.cc:729] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.003684 9913 consensus_queue.cc:260] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.003975 9914 raft_consensus.cc:348] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.004272 9913 ts_tablet_manager.cc:1398] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:50.004261 9914 raft_consensus.cc:374] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.004151 9912 consensus_queue.cc:260] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.004448 9914 raft_consensus.cc:729] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.004724 9914 consensus_queue.cc:260] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.005404 9914 ts_tablet_manager.cc:1398] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:50.005736 9914 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:50.007400 9914 tablet_bootstrap.cc:654] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.011854 9912 ts_tablet_manager.cc:1398] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:16:50.013435 9912 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:50.015518 9914 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:50.015777 9914 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:16:50.015929 9912 tablet_bootstrap.cc:654] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.016453 9920 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:50.016428 9914 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.016620 9920 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20240502 08:16:50.016660 9914 raft_consensus.cc:374] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.016772 9914 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.017020 9914 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.017511 9914 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:50.017683 9920 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.017992 9920 raft_consensus.cc:374] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.018417 9920 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.018735 9920 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.020165 9920 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:50.020879 9920 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:50.023701 9920 tablet_bootstrap.cc:654] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.024083 9912 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:50.024245 9912 ts_tablet_manager.cc:1367] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:16:50.025010 9912 raft_consensus.cc:348] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.025238 9912 raft_consensus.cc:374] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.025338 9912 raft_consensus.cc:729] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.025557 9920 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:50.025585 9912 consensus_queue.cc:260] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.025712 9920 ts_tablet_manager.cc:1367] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:50.026160 9912 ts_tablet_manager.cc:1398] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:50.026428 9920 raft_consensus.cc:348] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.026657 9920 raft_consensus.cc:374] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.026736 9920 raft_consensus.cc:729] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.026965 9920 consensus_queue.cc:260] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.027592 9920 ts_tablet_manager.cc:1398] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:50.111171 9842 raft_consensus.cc:482] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.111413 9842 raft_consensus.cc:504] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.112033 9842 leader_election.cc:288] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:50.112630 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9479bf73bb1400c83c247f3654bb79b" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:50.112632 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9479bf73bb1400c83c247f3654bb79b" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:50.112859 8912 raft_consensus.cc:2454] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:50.112918 8650 raft_consensus.cc:2454] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:50.113374 8715 leader_election.cc:302] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters:
I20240502 08:16:50.113683 9842 raft_consensus.cc:2781] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.113777 9842 raft_consensus.cc:482] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.113859 9842 raft_consensus.cc:3037] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.116068 9842 raft_consensus.cc:504] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.116722 9842 leader_election.cc:288] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:50.117200 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9479bf73bb1400c83c247f3654bb79b" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:50.117262 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9479bf73bb1400c83c247f3654bb79b" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:50.117419 8650 raft_consensus.cc:3037] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.117491 8912 raft_consensus.cc:3037] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.119393 8650 raft_consensus.cc:2454] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:50.119590 8912 raft_consensus.cc:2454] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:50.119833 8717 leader_election.cc:302] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:50.120126 9842 raft_consensus.cc:2781] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.120461 9842 raft_consensus.cc:686] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:50.120739 9842 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:50.123857 8501 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:50.268800 9663 raft_consensus.cc:482] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.269016 9663 raft_consensus.cc:504] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.269810 9663 leader_election.cc:288] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:50.270561 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41873909d8db44f89580fb209502ab30" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:50.270867 8650 raft_consensus.cc:2454] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 0.
I20240502 08:16:50.271308 8848 leader_election.cc:302] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:50.271637 9663 raft_consensus.cc:2781] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.271785 9663 raft_consensus.cc:482] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.271889 9663 raft_consensus.cc:3037] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.274267 9663 raft_consensus.cc:504] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.275049 9663 leader_election.cc:288] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:50.275610 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41873909d8db44f89580fb209502ab30" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:50.275817 8650 raft_consensus.cc:3037] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.277079 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41873909d8db44f89580fb209502ab30" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:50.277107 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41873909d8db44f89580fb209502ab30" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:50.277408 9043 raft_consensus.cc:3037] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.278190 8650 raft_consensus.cc:2454] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:50.278630 8848 leader_election.cc:302] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:50.278949 9663 raft_consensus.cc:2781] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.279407 9663 raft_consensus.cc:686] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Becoming Leader. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:16:50.279729 9663 consensus_queue.cc:237] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:50.279974 9043 raft_consensus.cc:2454] T 41873909d8db44f89580fb209502ab30 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:50.283123 8501 catalog_manager.cc:5484] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca reported cstate change: term changed from 0 to 1, leader changed from <none> to 71df2128b229478facd8ba1b2683ccca (127.8.68.195). New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:50.320010 9663 raft_consensus.cc:482] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.320204 9663 raft_consensus.cc:504] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.320822 9663 leader_election.cc:288] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:16:50.321384 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29f136434ad4ff1b0c9d4ac4481e216" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:50.321439 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29f136434ad4ff1b0c9d4ac4481e216" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:50.321632 8782 raft_consensus.cc:2454] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 0.
I20240502 08:16:50.321662 9043 raft_consensus.cc:2454] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 0.
I20240502 08:16:50.322081 8846 leader_election.cc:302] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:50.322409 9663 raft_consensus.cc:2781] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.322508 9663 raft_consensus.cc:482] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.322628 9663 raft_consensus.cc:3037] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.324633 9663 raft_consensus.cc:504] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.325182 9663 leader_election.cc:288] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:16:50.325527 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29f136434ad4ff1b0c9d4ac4481e216" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:50.325757 8782 raft_consensus.cc:3037] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.325690 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29f136434ad4ff1b0c9d4ac4481e216" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:50.325873 9043 raft_consensus.cc:3037] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.327872 8782 raft_consensus.cc:2454] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:50.328094 9043 raft_consensus.cc:2454] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:16:50.328317 8847 leader_election.cc:302] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters:
I20240502 08:16:50.328617 9663 raft_consensus.cc:2781] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.328802 9663 raft_consensus.cc:686] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Becoming Leader. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:16:50.329131 9663 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:16:50.332222 8501 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca reported cstate change: term changed from 0 to 1, leader changed from <none> to 71df2128b229478facd8ba1b2683ccca (127.8.68.195). New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:50.664712 9842 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.670964 9937 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:50.688969 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:50.689329 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:50.697772 9932 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:50.733049 9663 consensus_queue.cc:1035] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.737653 9663 consensus_queue.cc:1035] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 08:16:50.752336 9952 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.757428 9929 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 214.036 ms
time per row: 0.107018 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:16:52.024040 8503 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56274:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:52.034875 8629 tablet_service.cc:1467] Processing CreateTablet for tablet e606e24b29c04377ad5eb8576dc4c370 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.034855 8762 tablet_service.cc:1467] Processing CreateTablet for tablet e606e24b29c04377ad5eb8576dc4c370 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.035373 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e606e24b29c04377ad5eb8576dc4c370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.035503 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e606e24b29c04377ad5eb8576dc4c370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.036535 8891 tablet_service.cc:1467] Processing CreateTablet for tablet 531f3f63a6c0426987a56f9f8d5e0c56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.036659 9022 tablet_service.cc:1467] Processing CreateTablet for tablet e8896dce80734a7b85155855d6cb3303 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.036958 8891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531f3f63a6c0426987a56f9f8d5e0c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.037039 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8896dce80734a7b85155855d6cb3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.041965 9021 tablet_service.cc:1467] Processing CreateTablet for tablet 531f3f63a6c0426987a56f9f8d5e0c56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.042006 8630 tablet_service.cc:1467] Processing CreateTablet for tablet e8896dce80734a7b85155855d6cb3303 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.042368 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8896dce80734a7b85155855d6cb3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.042387 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531f3f63a6c0426987a56f9f8d5e0c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.045637 8893 tablet_service.cc:1467] Processing CreateTablet for tablet e606e24b29c04377ad5eb8576dc4c370 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.046061 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e606e24b29c04377ad5eb8576dc4c370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.047367 10007 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:52.048173 10006 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:52.048947 8760 tablet_service.cc:1467] Processing CreateTablet for tablet 531f3f63a6c0426987a56f9f8d5e0c56 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.049396 8760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531f3f63a6c0426987a56f9f8d5e0c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.049834 10006 tablet_bootstrap.cc:654] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.049914 10007 tablet_bootstrap.cc:654] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.048944 8761 tablet_service.cc:1467] Processing CreateTablet for tablet e8896dce80734a7b85155855d6cb3303 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.051704 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8896dce80734a7b85155855d6cb3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.058533 10007 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:52.058522 10006 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:52.058691 10006 ts_tablet_manager.cc:1367] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 08:16:52.058763 10007 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 08:16:52.059582 10006 raft_consensus.cc:348] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.059585 10007 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.059801 10007 raft_consensus.cc:374] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.059908 10007 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.059952 10006 raft_consensus.cc:374] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.060161 10006 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.060242 10007 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.060460 10006 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.060987 10006 ts_tablet_manager.cc:1398] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:52.062214 10007 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:52.063192 10007 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a: Bootstrap starting.
I20240502 08:16:52.065059 10007 tablet_bootstrap.cc:654] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.066790 10007 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a: No bootstrap required, opened a new log
I20240502 08:16:52.066962 10007 ts_tablet_manager.cc:1367] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:52.067716 10007 raft_consensus.cc:348] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.067904 10007 raft_consensus.cc:374] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.067987 10007 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.068229 10007 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.068706 10007 ts_tablet_manager.cc:1398] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:52.071115 10006 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca: Bootstrap starting.
I20240502 08:16:52.073313 10006 tablet_bootstrap.cc:654] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.073565 10010 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:52.075374 10010 tablet_bootstrap.cc:654] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.075366 10005 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:52.075871 10006 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca: No bootstrap required, opened a new log
I20240502 08:16:52.076045 10006 ts_tablet_manager.cc:1367] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:52.076840 10006 raft_consensus.cc:348] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.077008 10005 tablet_bootstrap.cc:654] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.077052 10006 raft_consensus.cc:374] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.077136 10006 raft_consensus.cc:729] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.077389 10006 consensus_queue.cc:260] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.077499 10010 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:52.077661 10010 ts_tablet_manager.cc:1367] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:16:52.078052 10006 ts_tablet_manager.cc:1398] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:52.078716 10010 raft_consensus.cc:348] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.079013 10010 raft_consensus.cc:374] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.079133 10010 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.079469 10010 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.083318 10010 ts_tablet_manager.cc:1398] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:52.084206 10010 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:52.084550 10005 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:52.084728 10005 ts_tablet_manager.cc:1367] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 08:16:52.085580 10005 raft_consensus.cc:348] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.085791 10005 raft_consensus.cc:374] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.085930 10005 raft_consensus.cc:729] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.086201 10005 consensus_queue.cc:260] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.086767 10005 ts_tablet_manager.cc:1398] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:52.087060 10010 tablet_bootstrap.cc:654] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.087172 10005 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:16:52.088670 10005 tablet_bootstrap.cc:654] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.089128 10010 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:52.089296 10010 ts_tablet_manager.cc:1367] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:16:52.090018 10010 raft_consensus.cc:348] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.090255 10010 raft_consensus.cc:374] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.090341 10010 raft_consensus.cc:729] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.090664 10005 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: No bootstrap required, opened a new log
I20240502 08:16:52.090602 10010 consensus_queue.cc:260] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.090811 10005 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:52.091189 10010 ts_tablet_manager.cc:1398] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:52.091593 10010 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Bootstrap starting.
I20240502 08:16:52.091701 10005 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.091922 10005 raft_consensus.cc:374] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.092000 10005 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.092252 10005 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.093218 10010 tablet_bootstrap.cc:654] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.093645 10005 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:52.095633 10010 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: No bootstrap required, opened a new log
I20240502 08:16:52.095772 10010 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:16:52.096429 10010 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.096591 10010 raft_consensus.cc:374] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.096684 10010 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.096889 10010 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.097584 10010 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:52.189157 9822 raft_consensus.cc:482] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.189365 9822 raft_consensus.cc:504] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.190104 9822 leader_election.cc:288] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:52.190747 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:52.190748 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:16:52.191004 8913 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 0.
I20240502 08:16:52.191043 8782 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 0.
I20240502 08:16:52.191560 8585 leader_election.cc:302] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:52.191839 9822 raft_consensus.cc:2781] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:52.191968 9822 raft_consensus.cc:482] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.192046 9822 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.193929 9822 raft_consensus.cc:504] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.194526 9822 leader_election.cc:288] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:16:52.195062 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:16:52.195155 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "bdf1a2f21da04697b58f19f331492b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:52.195276 8782 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.195411 8913 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.197430 8913 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 1.
I20240502 08:16:52.197460 8782 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf1a2f21da04697b58f19f331492b84 in term 1.
I20240502 08:16:52.198004 8585 leader_election.cc:302] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84; no voters:
I20240502 08:16:52.198544 9822 raft_consensus.cc:2781] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.199018 9822 raft_consensus.cc:686] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 LEADER]: Becoming Leader. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Running, Role: LEADER
I20240502 08:16:52.199378 9822 consensus_queue.cc:237] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:16:52.202381 8503 catalog_manager.cc:5484] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf1a2f21da04697b58f19f331492b84 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "bdf1a2f21da04697b58f19f331492b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:52.208292 9842 raft_consensus.cc:482] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.208523 9842 raft_consensus.cc:504] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.209210 9842 leader_election.cc:288] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747), 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:16:52.209714 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:16:52.209806 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:52.209929 8913 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:52.210067 9043 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:52.210387 8715 leader_election.cc:302] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters:
I20240502 08:16:52.210672 9842 raft_consensus.cc:2781] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:52.210775 9842 raft_consensus.cc:482] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.210884 9842 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.212778 9842 raft_consensus.cc:504] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.213339 9842 leader_election.cc:288] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747), 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:16:52.213840 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:52.213840 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:16:52.214072 9043 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.214073 8913 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.216943 9043 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:52.216972 8913 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:52.217411 8715 leader_election.cc:302] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:52.217720 9842 raft_consensus.cc:2781] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.218215 9842 raft_consensus.cc:686] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:52.218529 9842 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:16:52.221143 8503 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:52.416237 9842 raft_consensus.cc:482] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.416455 9842 raft_consensus.cc:504] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.417093 9842 leader_election.cc:288] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:52.417662 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84" is_pre_election: true
I20240502 08:16:52.417706 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a" is_pre_election: true
I20240502 08:16:52.417932 9043 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:52.417912 8650 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 0.
I20240502 08:16:52.418397 8715 leader_election.cc:302] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:52.418695 9842 raft_consensus.cc:2781] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:52.418833 9842 raft_consensus.cc:482] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.418936 9842 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.420907 9842 raft_consensus.cc:504] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.421571 9842 leader_election.cc:288] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:16:52.422077 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:16:52.422128 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf1a2f21da04697b58f19f331492b84"
I20240502 08:16:52.422317 9043 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.422345 8650 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.424499 9043 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:52.424973 8715 leader_election.cc:302] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:16:52.425145 8650 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
I20240502 08:16:52.425318 9842 raft_consensus.cc:2781] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.425489 9842 raft_consensus.cc:686] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:16:52.425788 9842 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:16:52.428972 8503 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:52.585300 9822 consensus_queue.cc:1035] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.605448 10027 consensus_queue.cc:1035] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.692524 9842 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.697085 9842 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:52.876091 10023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:52.876439 10023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:52.884804 10023 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:16:52.928664 9842 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.934376 9842 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 277.19 ms
time per row: 0.138595 ms
I20240502 08:16:53.333317 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:37561
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:40717
--webserver_port=39735
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 08:16:53.694023 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:53.694342 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:53.694476 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:53.703083 10086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:53.703269 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:53.703460 10086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:16:53.714149 10086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:37561
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=39735
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:53.715778 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:53.717615 10086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:53.731832 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:53.732412 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:53.733213 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:53.733978 10086 server_base.cc:1034] running on GCE node
I20240502 08:16:53.734648 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:53.735734 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:53.736928 10086 hybrid_clock.cc:648] HybridClock initialized: now 1714637813736848 us; error 79 us; skew 500 ppm
I20240502 08:16:53.737263 10086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:53.739804 10086 webserver.cc:460] Webserver started at http://127.8.68.197:39735/ using document root <none> and password file <none>
I20240502 08:16:53.740597 10086 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:53.740736 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:53.746580 10086 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:16:53.749676 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:53.751502 10086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:53.751665 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1f0504c982c42b8b9de29b8830c1e82"
format_stamp: "Formatted at 2024-05-02 08:16:38 on dist-test-slave-2dhs"
I20240502 08:16:53.753062 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:53.784651 10086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:53.785550 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:53.785847 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:53.786887 10086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:53.788578 10086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:53.788661 10086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:53.788800 10086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:53.788852 10086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:53.822552 10086 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:37561
I20240502 08:16:53.822705 10212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:37561 every 8 connection(s)
I20240502 08:16:53.824208 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:53.833389 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 10086
I20240502 08:16:53.834159 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:36779
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:40717
--webserver_port=40561
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:53.838279 10213 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:53.838618 10213 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:53.839768 10213 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:53.841559 8507 ts_manager.cc:194] Re-registered known tserver with Master: d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561)
I20240502 08:16:53.842474 8507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:41499
W20240502 08:16:54.199327 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:54.199641 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:54.199779 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:54.208220 10217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:54.208346 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:54.208499 10217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:16:54.218417 10217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:36779
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=40561
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:54.220252 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:54.222075 10217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:54.235808 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:54.236611 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:54.237439 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:54.238296 10217 server_base.cc:1034] running on GCE node
I20240502 08:16:54.238976 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:54.240033 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:54.241278 10217 hybrid_clock.cc:648] HybridClock initialized: now 1714637814241222 us; error 47 us; skew 500 ppm
I20240502 08:16:54.241636 10217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:54.244256 10217 webserver.cc:460] Webserver started at http://127.8.68.198:40561/ using document root <none> and password file <none>
I20240502 08:16:54.245061 10217 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:54.245203 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:54.250797 10217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:54.254045 10231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.255739 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 08:16:54.255925 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4586652e013946998246ca5489ae7083"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:54.256780 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:54.281461 10217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:54.282327 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:54.282617 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:54.283756 10217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:54.285298 10217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:54.285382 10217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.285518 10217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:54.286062 10217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.320925 10217 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:36779
I20240502 08:16:54.320969 10343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:36779 every 8 connection(s)
I20240502 08:16:54.322576 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:54.326743 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 10217
I20240502 08:16:54.327615 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:45071
--local_ip_for_outbound_sockets=127.8.68.199
--tserver_master_addrs=127.8.68.254:40717
--webserver_port=42939
--webserver_interface=127.8.68.199
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:54.338826 10344 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:54.339157 10344 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:54.340073 10344 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:54.341606 8507 ts_manager.cc:194] Re-registered known tserver with Master: 4586652e013946998246ca5489ae7083 (127.8.68.198:36779)
I20240502 08:16:54.342686 8507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:36031
W20240502 08:16:54.718137 10347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:54.718505 10347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:54.718714 10347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:54.728121 10347 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:54.728292 10347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:54.728483 10347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:16:54.739351 10347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:45071
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=42939
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:54.740818 10347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:54.742935 10347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:54.757508 10352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:54.758338 10353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:54.759186 10355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:54.759738 10347 server_base.cc:1034] running on GCE node
I20240502 08:16:54.760402 10347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:54.761495 10347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:54.762707 10347 hybrid_clock.cc:648] HybridClock initialized: now 1714637814762637 us; error 44 us; skew 500 ppm
I20240502 08:16:54.763072 10347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:54.765961 10347 webserver.cc:460] Webserver started at http://127.8.68.199:42939/ using document root <none> and password file <none>
I20240502 08:16:54.766954 10347 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:54.767112 10347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:54.772996 10347 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 08:16:54.776199 10361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.777786 10347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 08:16:54.777995 10347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eae55bf90c524e8eb9df4c31f40096e4"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-2dhs"
I20240502 08:16:54.778882 10347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:54.815733 10347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:54.817040 10347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:54.817466 10347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:54.818890 10347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:54.821352 10347 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:54.821455 10347 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.821610 10347 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:54.821681 10347 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:54.845350 10213 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:54.878269 10347 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:45071
I20240502 08:16:54.878325 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:45071 every 8 connection(s)
I20240502 08:16:54.879985 10347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:16:54.889221 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 10347
I20240502 08:16:54.889900 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:32925
--local_ip_for_outbound_sockets=127.8.68.200
--tserver_master_addrs=127.8.68.254:40717
--webserver_port=45707
--webserver_interface=127.8.68.200
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 08:16:54.894515 10474 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:54.894919 10474 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:54.895972 10474 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:54.897553 8507 ts_manager.cc:194] Re-registered known tserver with Master: eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071)
I20240502 08:16:54.898918 8507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:44611
I20240502 08:16:55.346032 10344 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
W20240502 08:16:55.391305 10478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:55.391659 10478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:55.391834 10478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:55.401207 10478 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:55.401386 10478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:55.401569 10478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240502 08:16:55.412562 10478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:32925
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=45707
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:55.413996 10478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:55.415956 10478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:55.430968 10485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:55.431653 10484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:55.439695 10487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:55.448511 10478 server_base.cc:1034] running on GCE node
I20240502 08:16:55.449306 10478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:55.450573 10478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:55.451790 10478 hybrid_clock.cc:648] HybridClock initialized: now 1714637815451729 us; error 39 us; skew 500 ppm
I20240502 08:16:55.452169 10478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:55.455343 10478 webserver.cc:460] Webserver started at http://127.8.68.200:45707/ using document root <none> and password file <none>
I20240502 08:16:55.456192 10478 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:55.456321 10478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:55.464011 10478 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:16:55.467474 10493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:55.469146 10478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:55.469342 10478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-2dhs"
I20240502 08:16:55.470494 10478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:55.487373 10478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:55.488431 10478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:55.488790 10478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:55.489933 10478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:55.491885 10478 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:55.491977 10478 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:55.492149 10478 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:55.492223 10478 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:55.543009 10478 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:32925
I20240502 08:16:55.543048 10606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:32925 every 8 connection(s)
I20240502 08:16:55.544694 10478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:16:55.545739 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 10478
I20240502 08:16:55.561071 10607 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:16:55.561427 10607 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:55.562353 10607 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:16:55.563900 8507 ts_manager.cc:194] Re-registered known tserver with Master: 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925)
I20240502 08:16:55.564822 8507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:53137
I20240502 08:16:55.902388 10474 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:56.099489 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:56.123631 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:56.135077 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:56.138476 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:16:56.158326 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:56.167486 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:56.177094 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:56.199965 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:56.567842 10607 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:16:56.786875 8505 ts_manager.cc:292] Set tserver state for bdf1a2f21da04697b58f19f331492b84 to MAINTENANCE_MODE
W20240502 08:16:58.223024 10678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:58.223454 10678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:58.232468 10678 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:58.472831 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:58.490384 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.490881 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.522125 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:58.527511 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.541828 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:16:58.551301 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.563659 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:16:58.567589 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:58.603605 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.634209 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.635193 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:58.663053 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.665715 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.688812 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.702415 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.981940 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:59.012012 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.023947 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.047628 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:16:59.073029 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:59.078161 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.096110 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.109716 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:59.363919 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:59.386354 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.405316 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.415990 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:16:59.447325 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.461583 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:59.462270 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:59.464066 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:16:59.553655 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:59.554108 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:59.566248 10798 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:59.581655 8782 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } }
I20240502 08:16:59.584426 8649 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:16:59.584456 9042 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:16:59.585026 10765 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:16:59.585665 10765 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:16:59.589017 10857 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:16:59.590250 8649 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:16:59.592303 9042 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:16:59.595921 8508 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:16:59.606825 8782 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:16:59.609779 8649 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:16:59.610392 10857 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:16:59.614377 10857 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:16:59.616881 9042 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:16:59.619689 10765 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:16:59.620723 8649 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:16:59.625409 8495 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 888fe8d0871a4a539b43569e0eda4bef with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:16:59.626865 8507 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:59.626766 9042 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
W20240502 08:16:59.637717 8718 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 -> Peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925): Couldn't send request to peer 4f6d482dba7d46b89e70150a61cb5cf0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 888fe8d0871a4a539b43569e0eda4bef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:59.730973 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:16:59.744438 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.752137 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:16:59.753700 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.755399 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.770568 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:59.789623 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:59.789819 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:00.194726 10911 ts_tablet_manager.cc:902] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:00.196766 10911 tablet_copy_client.cc:308] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:00.204449 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet 888fe8d0871a4a539b43569e0eda4bef from peer 4f6d482dba7d46b89e70150a61cb5cf0 ({username='slave'} at 127.8.68.200:34981)
I20240502 08:17:00.204689 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet 888fe8d0871a4a539b43569e0eda4bef from peer 4f6d482dba7d46b89e70150a61cb5cf0 at {username='slave'} at 127.8.68.200:34981: session id = 4f6d482dba7d46b89e70150a61cb5cf0-888fe8d0871a4a539b43569e0eda4bef
I20240502 08:17:00.207319 8802 tablet_copy_source_session.cc:205] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:00.212152 10911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.223539 10911 tablet_copy_client.cc:783] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:00.223956 10911 tablet_copy_client.cc:647] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:00.226476 10915 tablet_copy_client.cc:1002] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:00.379588 10905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:00.379951 10905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:00.387944 10905 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:00.526540 8760 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:00.541927 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:00.570236 8893 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:00.588521 8629 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:00.589058 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:00.590636 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:00.602464 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:00.615833 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 08:17:00.793095 10676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 8569
I20240502 08:17:00.800956 8691 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:00.927376 10676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 8569
W20240502 08:17:00.950979 8848 connection.cc:535] server connection from 127.8.68.193:57461 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:00.968981 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:00.970512 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.106568 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:01.108106 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.108263 8717 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.122588 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.164002 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.176270 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.241859 8717 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.298422 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:01.299805 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.369160 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.727020 10911 tablet_copy_client.cc:523] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:01.730403 10911 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap starting.
I20240502 08:17:01.741497 10911 log.cc:824] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:01.773638 10911 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:01.774116 10911 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap complete.
I20240502 08:17:01.774534 10911 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent bootstrapping tablet: real 0.044s user 0.042s sys 0.001s
I20240502 08:17:01.779350 10911 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:01.779966 10911 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Initialized, Role: LEARNER
I20240502 08:17:01.780661 10911 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:01.782487 10911 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20240502 08:17:01.783869 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-888fe8d0871a4a539b43569e0eda4bef received from {username='slave'} at 127.8.68.200:34981
I20240502 08:17:01.784096 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-888fe8d0871a4a539b43569e0eda4bef on tablet 888fe8d0871a4a539b43569e0eda4bef with peer 4f6d482dba7d46b89e70150a61cb5cf0
I20240502 08:17:01.911058 11014 raft_consensus.cc:482] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:01.911334 11014 raft_consensus.cc:504] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:01.912083 11014 leader_election.cc:288] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:01.912765 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
W20240502 08:17:01.913427 8848 leader_election.cc:334] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:01.913666 8847 leader_election.cc:302] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca; no voters: 14ea87db2f6a4497a2a961afecd23a32, bdf1a2f21da04697b58f19f331492b84
I20240502 08:17:01.913925 11014 raft_consensus.cc:2726] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:01.916168 10562 raft_consensus.cc:1203] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:17:02.162007 11012 raft_consensus.cc:482] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:02.162222 11012 raft_consensus.cc:504] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:02.163206 11012 leader_election.cc:288] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:02.163832 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:17:02.164171 8911 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 1.
W20240502 08:17:02.164615 8717 leader_election.cc:334] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.164809 8715 leader_election.cc:302] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters: bdf1a2f21da04697b58f19f331492b84
I20240502 08:17:02.165087 11012 raft_consensus.cc:2781] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:17:02.165202 11012 raft_consensus.cc:482] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:02.165274 11012 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:02.167300 11012 raft_consensus.cc:504] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:02.167819 11012 leader_election.cc:288] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 election: Requested vote from peers bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415), 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:02.168399 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606e24b29c04377ad5eb8576dc4c370" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:17:02.168623 8911 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:17:02.169098 8717 leader_election.cc:334] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.170684 8911 raft_consensus.cc:2454] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 2.
I20240502 08:17:02.171125 8715 leader_election.cc:302] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters: bdf1a2f21da04697b58f19f331492b84
I20240502 08:17:02.171424 11012 raft_consensus.cc:2781] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:02.171582 11012 raft_consensus.cc:686] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:17:02.171844 11012 consensus_queue.cc:237] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:02.173491 11012 raft_consensus.cc:482] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:02.173646 11012 raft_consensus.cc:504] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:02.174257 11012 leader_election.cc:288] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:17:02.174669 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "71df2128b229478facd8ba1b2683ccca" is_pre_election: true
I20240502 08:17:02.174767 8507 catalog_manager.cc:5484] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 1 to 2, leader changed from bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 2 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.175617 8717 leader_election.cc:334] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.175809 8717 leader_election.cc:302] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32; no voters: 71df2128b229478facd8ba1b2683ccca, bdf1a2f21da04697b58f19f331492b84
I20240502 08:17:02.176178 11012 raft_consensus.cc:2726] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:02.343413 11012 raft_consensus.cc:1050] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 to VOTER
I20240502 08:17:02.344209 11012 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:02.346935 10562 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:02.347589 9042 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:02.347697 11038 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:17:02.348022 8717 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.348220 10999 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:02.353111 10999 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.353955 10562 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.357564 9042 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.359040 8507 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.365953 8781 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:02.367902 10562 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:02.368305 9042 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:02.368846 11041 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:02.369280 10999 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:02.371371 11041 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.372010 9042 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.372025 10562 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:02.376942 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 888fe8d0871a4a539b43569e0eda4bef with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:02.376745 8509 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
W20240502 08:17:02.381103 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:02.382972 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 888fe8d0871a4a539b43569e0eda4bef: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.428053 11014 raft_consensus.cc:482] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:02.428236 11014 raft_consensus.cc:504] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:02.428891 11014 leader_election.cc:288] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:17:02.429471 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32" is_pre_election: true
I20240502 08:17:02.429708 8781 raft_consensus.cc:2454] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 1.
I20240502 08:17:02.430101 8847 leader_election.cc:302] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters:
W20240502 08:17:02.430361 8848 leader_election.cc:334] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.430665 11014 raft_consensus.cc:2781] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:17:02.430774 11014 raft_consensus.cc:482] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdf1a2f21da04697b58f19f331492b84)
I20240502 08:17:02.430858 11014 raft_consensus.cc:3037] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:02.433130 11014 raft_consensus.cc:504] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:02.434242 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49af926b61e64eb0ac9c4dcd5b638152" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:17:02.434480 8781 raft_consensus.cc:3037] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:17:02.435274 8848 leader_election.cc:334] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:02.436084 11014 leader_election.cc:288] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:17:02.437417 8781 raft_consensus.cc:2454] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 2.
I20240502 08:17:02.437834 8847 leader_election.cc:302] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca; no voters: bdf1a2f21da04697b58f19f331492b84
I20240502 08:17:02.438124 11014 raft_consensus.cc:2781] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:02.438557 11014 raft_consensus.cc:686] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Becoming Leader. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:17:02.438933 11014 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:02.441768 8507 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca reported cstate change: term changed from 1 to 2, leader changed from bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) to 71df2128b229478facd8ba1b2683ccca (127.8.68.195). New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.597277 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.621601 8911 raft_consensus.cc:1261] T e606e24b29c04377ad5eb8576dc4c370 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:17:02.622241 11041 consensus_queue.cc:1035] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:02.732640 8781 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } }
I20240502 08:17:02.734807 9042 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:02.735394 11012 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:02.735749 8717 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.737924 11041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:17:02.738508 9042 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:17:02.743173 8507 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.747875 8781 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:02.750703 9042 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:02.751461 11012 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:02.754123 11041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:02.755045 9042 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:02.760820 8495 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e8896dce80734a7b85155855d6cb3303 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:17:02.760660 8507 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 4 to 5, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
W20240502 08:17:02.763763 8715 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer 4586652e013946998246ca5489ae7083 (127.8.68.198:36779): Couldn't send request to peer 4586652e013946998246ca5489ae7083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8896dce80734a7b85155855d6cb3303. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:02.768988 8717 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.833390 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:17:02.834026 11014 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:02.839695 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:03.175741 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:03.269323 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:03.272755 11080 ts_tablet_manager.cc:902] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:03.274374 11080 tablet_copy_client.cc:308] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:03.280627 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet e8896dce80734a7b85155855d6cb3303 from peer 4586652e013946998246ca5489ae7083 ({username='slave'} at 127.8.68.198:45733)
I20240502 08:17:03.280803 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet e8896dce80734a7b85155855d6cb3303 from peer 4586652e013946998246ca5489ae7083 at {username='slave'} at 127.8.68.198:45733: session id = 4586652e013946998246ca5489ae7083-e8896dce80734a7b85155855d6cb3303
I20240502 08:17:03.282733 8802 tablet_copy_source_session.cc:205] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.287189 11080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8896dce80734a7b85155855d6cb3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.297968 11080 tablet_copy_client.cc:783] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.298310 11080 tablet_copy_client.cc:647] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.300263 11084 tablet_copy_client.cc:1002] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:03.598093 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 888fe8d0871a4a539b43569e0eda4bef: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:03.639321 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:03.672966 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:03.680702 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:03.721238 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:03.743180 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:04.667649 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 888fe8d0871a4a539b43569e0eda4bef: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:04.800858 11080 tablet_copy_client.cc:523] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:04.804330 11080 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Bootstrap starting.
I20240502 08:17:04.813881 11080 log.cc:824] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:04.845849 11080 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:04.846395 11080 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Bootstrap complete.
I20240502 08:17:04.846796 11080 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.000s
I20240502 08:17:04.851179 11080 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:04.851604 11080 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4586652e013946998246ca5489ae7083, State: Initialized, Role: LEARNER
I20240502 08:17:04.852092 11080 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:04.853662 11080 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 08:17:04.854604 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session 4586652e013946998246ca5489ae7083-e8896dce80734a7b85155855d6cb3303 received from {username='slave'} at 127.8.68.198:45733
I20240502 08:17:04.854743 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session 4586652e013946998246ca5489ae7083-e8896dce80734a7b85155855d6cb3303 on tablet e8896dce80734a7b85155855d6cb3303 with peer 4586652e013946998246ca5489ae7083
W20240502 08:17:04.961773 8717 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:05.132805 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:17:05.189766 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:05.313349 10299 raft_consensus.cc:1203] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 08:17:05.700376 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:05.719491 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:17:05.774695 11155 raft_consensus.cc:1050] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER 4586652e013946998246ca5489ae7083 to VOTER
I20240502 08:17:05.775513 11155 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:05.778024 9041 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:05.778065 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:05.778677 11154 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.779021 11155 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:17:05.779341 8717 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:05.782680 11155 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.783483 10299 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.787230 9041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.788375 8507 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 5 to 6, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.794672 8781 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:05.796562 9041 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:05.796603 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:05.797240 11157 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:05.797528 11154 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:05.799891 11155 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.800609 9041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.800563 10299 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:05.804131 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8896dce80734a7b85155855d6cb3303 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:05.805410 8509 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
W20240502 08:17:05.812487 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:05.867748 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:05.920387 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:06.009196 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:17:06.046204 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:17:06.137876 9041 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:17:06.140363 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:06.140938 11178 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:06.141458 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:06.143895 11005 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } }
I20240502 08:17:06.144558 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } }
I20240502 08:17:06.149128 8507 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:06.155015 9041 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:06.158756 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:06.159539 11005 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:06.162389 11005 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:06.162813 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:06.167586 8494 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 25df2da40e524e9c95cbe13a484445b5 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 08:17:06.167846 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:06.169034 8508 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 4 to 5, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:06.179001 8979 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925): Couldn't send request to peer 4f6d482dba7d46b89e70150a61cb5cf0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25df2da40e524e9c95cbe13a484445b5. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:06.243016 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 08:17:06.419502 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:17:06.556275 11192 ts_tablet_manager.cc:902] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Initiating tablet copy from peer 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:17:06.557713 11192 tablet_copy_client.cc:308] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:35907
I20240502 08:17:06.564461 9063 tablet_copy_service.cc:140] P 78c8d87aa5e44674959cd90af335de6a: Received BeginTabletCopySession request for tablet 25df2da40e524e9c95cbe13a484445b5 from peer 4f6d482dba7d46b89e70150a61cb5cf0 ({username='slave'} at 127.8.68.200:37429)
I20240502 08:17:06.564733 9063 tablet_copy_service.cc:161] P 78c8d87aa5e44674959cd90af335de6a: Beginning new tablet copy session on tablet 25df2da40e524e9c95cbe13a484445b5 from peer 4f6d482dba7d46b89e70150a61cb5cf0 at {username='slave'} at 127.8.68.200:37429: session id = 4f6d482dba7d46b89e70150a61cb5cf0-25df2da40e524e9c95cbe13a484445b5
I20240502 08:17:06.567487 9063 tablet_copy_source_session.cc:205] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:06.569355 11192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.573951 11192 tablet_copy_client.cc:783] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:06.574324 11192 tablet_copy_client.cc:647] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:06.576450 11196 tablet_copy_client.cc:1002] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:06.581465 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 08:17:06.582708 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:06.956570 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:07.624174 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:07.731935 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:17:08.022783 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 08:17:08.024055 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:08.076855 11192 tablet_copy_client.cc:523] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:08.079715 11192 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap starting.
I20240502 08:17:08.111670 11192 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:08.112105 11192 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap complete.
I20240502 08:17:08.112396 11192 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20240502 08:17:08.113257 11192 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:08.113471 11192 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Initialized, Role: LEARNER
I20240502 08:17:08.113672 11192 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:08.114430 11192 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:08.115582 9063 tablet_copy_service.cc:342] P 78c8d87aa5e44674959cd90af335de6a: Request end of tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-25df2da40e524e9c95cbe13a484445b5 received from {username='slave'} at 127.8.68.200:37429
I20240502 08:17:08.115770 9063 tablet_copy_service.cc:433] P 78c8d87aa5e44674959cd90af335de6a: ending tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-25df2da40e524e9c95cbe13a484445b5 on tablet 25df2da40e524e9c95cbe13a484445b5 with peer 4f6d482dba7d46b89e70150a61cb5cf0
W20240502 08:17:08.166846 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:08.280644 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:08.326166 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:08.342792 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:08.463819 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:08.483762 10562 raft_consensus.cc:1203] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:17:08.928972 11263 raft_consensus.cc:1050] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: attempting to promote NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 to VOTER
I20240502 08:17:08.930009 11263 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:08.932467 10562 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:08.933142 11263 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:08.933215 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 08:17:08.933825 8978 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:08.933830 11263 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:08.936664 11269 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.937299 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.941397 10562 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.942281 8507 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 5 to 6, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:08.949455 9041 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:08.951565 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:08.951648 10562 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:08.952240 11271 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:08.953194 11271 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:08.954566 11227 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.956131 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.957819 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25df2da40e524e9c95cbe13a484445b5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:08.958567 10562 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:08.960304 8509 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:09.064530 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:09.078547 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25df2da40e524e9c95cbe13a484445b5: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:09.150574 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:17:09.329983 8912 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } }
I20240502 08:17:09.332581 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:09.333231 11293 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:17:09.333551 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:09.335952 11014 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:17:09.336659 8781 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:17:09.342018 8507 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } }
I20240502 08:17:09.352864 8912 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:09.355712 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:09.356865 11014 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:17:09.357880 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:09.360332 11295 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:09.361111 8781 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:09.366190 8494 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 49af926b61e64eb0ac9c4dcd5b638152 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:17:09.367178 8509 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 5 to 6, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:09.368924 8849 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49af926b61e64eb0ac9c4dcd5b638152. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:09.895176 11314 ts_tablet_manager.cc:902] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Initiating tablet copy from peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:09.896781 11314 tablet_copy_client.cc:308] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:36747
I20240502 08:17:09.903043 8933 tablet_copy_service.cc:140] P 71df2128b229478facd8ba1b2683ccca: Received BeginTabletCopySession request for tablet 49af926b61e64eb0ac9c4dcd5b638152 from peer eae55bf90c524e8eb9df4c31f40096e4 ({username='slave'} at 127.8.68.199:33243)
I20240502 08:17:09.903259 8933 tablet_copy_service.cc:161] P 71df2128b229478facd8ba1b2683ccca: Beginning new tablet copy session on tablet 49af926b61e64eb0ac9c4dcd5b638152 from peer eae55bf90c524e8eb9df4c31f40096e4 at {username='slave'} at 127.8.68.199:33243: session id = eae55bf90c524e8eb9df4c31f40096e4-49af926b61e64eb0ac9c4dcd5b638152
I20240502 08:17:09.905891 8933 tablet_copy_source_session.cc:205] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:09.910437 11314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49af926b61e64eb0ac9c4dcd5b638152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:09.921275 11314 tablet_copy_client.cc:783] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:09.921605 11314 tablet_copy_client.cc:647] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:09.923753 11318 tablet_copy_client.cc:1002] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:10.101191 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:10.109959 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:17:10.608260 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:10.622974 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:10.720052 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:10.775235 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:11.179143 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25df2da40e524e9c95cbe13a484445b5: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:11.232972 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 08:17:11.424345 11314 tablet_copy_client.cc:523] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:11.429255 11314 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap starting.
I20240502 08:17:11.442231 11314 log.cc:824] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:11.479585 11314 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:11.480077 11314 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap complete.
I20240502 08:17:11.480587 11314 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Time spent bootstrapping tablet: real 0.052s user 0.038s sys 0.013s
I20240502 08:17:11.485426 11314 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:11.485810 11314 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eae55bf90c524e8eb9df4c31f40096e4, State: Initialized, Role: LEARNER
I20240502 08:17:11.486331 11314 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:11.488052 11314 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240502 08:17:11.488988 8933 tablet_copy_service.cc:342] P 71df2128b229478facd8ba1b2683ccca: Request end of tablet copy session eae55bf90c524e8eb9df4c31f40096e4-49af926b61e64eb0ac9c4dcd5b638152 received from {username='slave'} at 127.8.68.199:33243
I20240502 08:17:11.489172 8933 tablet_copy_service.cc:433] P 71df2128b229478facd8ba1b2683ccca: ending tablet copy session eae55bf90c524e8eb9df4c31f40096e4-49af926b61e64eb0ac9c4dcd5b638152 on tablet 49af926b61e64eb0ac9c4dcd5b638152 with peer eae55bf90c524e8eb9df4c31f40096e4
W20240502 08:17:11.512425 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 08:17:11.611797 10429 raft_consensus.cc:1203] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240502 08:17:11.639282 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:17:11.640300 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:11.701803 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:17:11.905398 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:12.197656 11387 raft_consensus.cc:1050] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: attempting to promote NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 to VOTER
I20240502 08:17:12.198390 11387 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:12.201710 10429 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 LEARNER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:17:12.201710 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:17:12.202374 11387 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:17:12.202677 8848 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:12.202746 11389 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:12.206284 11389 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.207093 10429 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.209923 8781 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.210980 8507 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 6 to 7, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:12.218622 8911 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:12.220429 10429 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:17:12.220471 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:17:12.221041 11333 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:12.221287 11387 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:12.223477 11333 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.224113 10429 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.224656 8781 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:12.227629 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49af926b61e64eb0ac9c4dcd5b638152 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:12.228513 8508 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 reported cstate change: config changed from index 7 to 8, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
W20240502 08:17:12.236315 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:12.516119 8781 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:12.518168 8911 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:12.518754 11411 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:12.519409 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:12.521580 11308 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } }
I20240502 08:17:12.522150 8911 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } }
I20240502 08:17:12.526440 8507 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:12.531632 8781 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:12.538688 8912 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:12.539597 11308 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:12.542404 11308 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
W20240502 08:17:12.551513 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:12.551892 8716 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9479bf73bb1400c83c247f3654bb79b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:12.552631 8495 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c9479bf73bb1400c83c247f3654bb79b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:17:12.553812 8912 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:12.554343 8509 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 4 to 5, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:12.874346 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:17:12.970319 11427 ts_tablet_manager.cc:902] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:12.972396 11427 tablet_copy_client.cc:308] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
W20240502 08:17:12.978519 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:17:12.982381 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet c9479bf73bb1400c83c247f3654bb79b from peer d1f0504c982c42b8b9de29b8830c1e82 ({username='slave'} at 127.8.68.197:59271)
I20240502 08:17:12.982599 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet c9479bf73bb1400c83c247f3654bb79b from peer d1f0504c982c42b8b9de29b8830c1e82 at {username='slave'} at 127.8.68.197:59271: session id = d1f0504c982c42b8b9de29b8830c1e82-c9479bf73bb1400c83c247f3654bb79b
I20240502 08:17:12.986470 8802 tablet_copy_source_session.cc:205] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:12.991845 11427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9479bf73bb1400c83c247f3654bb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:13.002239 11427 tablet_copy_client.cc:783] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:13.002560 11427 tablet_copy_client.cc:647] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:13.004510 11431 tablet_copy_client.cc:1002] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:13.241842 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:13.244535 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:13.252013 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 08:17:13.252914 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25df2da40e524e9c95cbe13a484445b5: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:14.000990 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:14.181865 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:14.413617 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:14.505117 11427 tablet_copy_client.cc:523] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:14.508666 11427 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap starting.
I20240502 08:17:14.518008 11427 log.cc:824] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:14.549299 11427 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:14.549818 11427 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap complete.
I20240502 08:17:14.550247 11427 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.005s
I20240502 08:17:14.555058 11427 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:14.555464 11427 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f0504c982c42b8b9de29b8830c1e82, State: Initialized, Role: LEARNER
I20240502 08:17:14.556015 11427 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:14.557766 11427 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 08:17:14.558722 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session d1f0504c982c42b8b9de29b8830c1e82-c9479bf73bb1400c83c247f3654bb79b received from {username='slave'} at 127.8.68.197:59271
I20240502 08:17:14.558842 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session d1f0504c982c42b8b9de29b8830c1e82-c9479bf73bb1400c83c247f3654bb79b on tablet c9479bf73bb1400c83c247f3654bb79b with peer d1f0504c982c42b8b9de29b8830c1e82
I20240502 08:17:14.889847 10168 raft_consensus.cc:1203] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 08:17:15.300601 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:17:15.359390 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:15.404413 11502 raft_consensus.cc:1050] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 to VOTER
I20240502 08:17:15.405341 11502 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:15.408421 10168 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:15.409132 8912 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:15.409241 11504 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:15.410200 11502 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240502 08:17:15.410725 8717 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:15.416939 11504 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.417904 10168 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.420928 8912 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.422946 8509 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 5 to 6, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:15.429627 8781 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:15.431764 10168 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:15.431758 8912 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:15.432437 11501 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:15.433357 11501 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:15.435045 11504 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.436651 10168 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.439285 8912 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.441989 8508 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 reported cstate change: config changed from index 6 to 7, VOTER bdf1a2f21da04697b58f19f331492b84 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:15.444228 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9479bf73bb1400c83c247f3654bb79b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 08:17:15.453496 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c9479bf73bb1400c83c247f3654bb79b: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:15.465054 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:17:15.523397 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:17:15.601631 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:17:15.830747 8760 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:15.849464 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:15.853955 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:15.876013 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:15.887146 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:15.944025 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:15.944545 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:16.117511 8760 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:16.126878 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:16.136740 10542 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:16.149260 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:16.152788 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:16.177331 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:16.208993 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:17:16.244757 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:17:16.276695 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:17:16.286617 8781 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:16.289033 10168 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:17:16.289122 8715 consensus_peers.cc:485] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 -> Peer 4586652e013946998246ca5489ae7083 (127.8.68.198:36779): Couldn't send request to peer 4586652e013946998246ca5489ae7083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9479bf73bb1400c83c247f3654bb79b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:16.289273 8912 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:16.289649 11504 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:16.289992 11599 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:16.292325 11448 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:16.292989 10168 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:16.293138 8912 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:16.299613 8503 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 7 to 8, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:16.320168 9042 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:16.323148 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:16.323642 10562 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:16.324318 11271 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:16.324664 11610 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:17:16.331597 8979 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25df2da40e524e9c95cbe13a484445b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:16.332890 11610 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:16.333799 10562 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:16.336508 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:16.338424 8509 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 7 to 8, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:16.356544 8912 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:16.359783 10429 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:16.360113 8781 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:16.360508 11386 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:16.360921 11593 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:16.365296 11593 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:16.366184 8781 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
W20240502 08:17:16.370961 8847 consensus_peers.cc:485] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49af926b61e64eb0ac9c4dcd5b638152. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:16.372326 8509 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 8 to 9, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:16.383563 10429 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
W20240502 08:17:16.470324 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:16.548599 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:17:16.654920 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 08:17:16.733246 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:17:16.765787 11654 ts_tablet_manager.cc:902] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:16.767408 11654 tablet_copy_client.cc:308] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:16.768337 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet c9479bf73bb1400c83c247f3654bb79b from peer 4586652e013946998246ca5489ae7083 ({username='slave'} at 127.8.68.198:45733)
I20240502 08:17:16.768538 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet c9479bf73bb1400c83c247f3654bb79b from peer 4586652e013946998246ca5489ae7083 at {username='slave'} at 127.8.68.198:45733: session id = 4586652e013946998246ca5489ae7083-c9479bf73bb1400c83c247f3654bb79b
I20240502 08:17:16.770627 8802 tablet_copy_source_session.cc:205] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:16.771962 11654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9479bf73bb1400c83c247f3654bb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.777097 11654 tablet_copy_client.cc:783] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:16.777382 11654 tablet_copy_client.cc:647] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:16.779281 11658 tablet_copy_client.cc:1002] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:16.887132 11674 ts_tablet_manager.cc:902] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Initiating tablet copy from peer 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:17:16.889811 11674 tablet_copy_client.cc:308] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:35907
I20240502 08:17:16.897033 9063 tablet_copy_service.cc:140] P 78c8d87aa5e44674959cd90af335de6a: Received BeginTabletCopySession request for tablet 25df2da40e524e9c95cbe13a484445b5 from peer eae55bf90c524e8eb9df4c31f40096e4 ({username='slave'} at 127.8.68.199:37207)
I20240502 08:17:16.897243 9063 tablet_copy_service.cc:161] P 78c8d87aa5e44674959cd90af335de6a: Beginning new tablet copy session on tablet 25df2da40e524e9c95cbe13a484445b5 from peer eae55bf90c524e8eb9df4c31f40096e4 at {username='slave'} at 127.8.68.199:37207: session id = eae55bf90c524e8eb9df4c31f40096e4-25df2da40e524e9c95cbe13a484445b5
I20240502 08:17:16.899370 9063 tablet_copy_source_session.cc:205] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:16.900800 11674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.905282 11674 tablet_copy_client.cc:783] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:16.905540 11674 tablet_copy_client.cc:647] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:16.907392 11678 tablet_copy_client.cc:1002] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:16.966013 11681 ts_tablet_manager.cc:902] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Initiating tablet copy from peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:16.967690 11681 tablet_copy_client.cc:308] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:36747
I20240502 08:17:16.975425 8933 tablet_copy_service.cc:140] P 71df2128b229478facd8ba1b2683ccca: Received BeginTabletCopySession request for tablet 49af926b61e64eb0ac9c4dcd5b638152 from peer d1f0504c982c42b8b9de29b8830c1e82 ({username='slave'} at 127.8.68.197:53343)
I20240502 08:17:16.975667 8933 tablet_copy_service.cc:161] P 71df2128b229478facd8ba1b2683ccca: Beginning new tablet copy session on tablet 49af926b61e64eb0ac9c4dcd5b638152 from peer d1f0504c982c42b8b9de29b8830c1e82 at {username='slave'} at 127.8.68.197:53343: session id = d1f0504c982c42b8b9de29b8830c1e82-49af926b61e64eb0ac9c4dcd5b638152
I20240502 08:17:16.977903 8933 tablet_copy_source_session.cc:205] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:16.979663 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49af926b61e64eb0ac9c4dcd5b638152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.987102 11681 tablet_copy_client.cc:783] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:16.987406 11681 tablet_copy_client.cc:647] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:16.989384 11685 tablet_copy_client.cc:1002] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:17.684026 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c9479bf73bb1400c83c247f3654bb79b: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:17.817054 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:17:17.914314 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:17:18.060775 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:17:18.261900 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:17:18.279736 11654 tablet_copy_client.cc:523] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:18.282687 11654 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Bootstrap starting.
I20240502 08:17:18.320912 11654 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:18.321344 11654 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Bootstrap complete.
I20240502 08:17:18.321621 11654 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.012s
I20240502 08:17:18.322450 11654 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:18.322695 11654 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4586652e013946998246ca5489ae7083, State: Initialized, Role: LEARNER
I20240502 08:17:18.322885 11654 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:18.323742 11654 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:17:18.324518 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session 4586652e013946998246ca5489ae7083-c9479bf73bb1400c83c247f3654bb79b received from {username='slave'} at 127.8.68.198:45733
I20240502 08:17:18.324644 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session 4586652e013946998246ca5489ae7083-c9479bf73bb1400c83c247f3654bb79b on tablet c9479bf73bb1400c83c247f3654bb79b with peer 4586652e013946998246ca5489ae7083
I20240502 08:17:18.407779 11674 tablet_copy_client.cc:523] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:18.410542 11674 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap starting.
I20240502 08:17:18.415154 10299 raft_consensus.cc:1203] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:17:18.453529 11674 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:18.453951 11674 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap complete.
I20240502 08:17:18.454257 11674 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Time spent bootstrapping tablet: real 0.044s user 0.035s sys 0.008s
I20240502 08:17:18.455058 11674 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:18.455291 11674 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eae55bf90c524e8eb9df4c31f40096e4, State: Initialized, Role: LEARNER
I20240502 08:17:18.455456 11674 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:18.456152 11674 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 08:17:18.457007 9063 tablet_copy_service.cc:342] P 78c8d87aa5e44674959cd90af335de6a: Request end of tablet copy session eae55bf90c524e8eb9df4c31f40096e4-25df2da40e524e9c95cbe13a484445b5 received from {username='slave'} at 127.8.68.199:37207
I20240502 08:17:18.457154 9063 tablet_copy_service.cc:433] P 78c8d87aa5e44674959cd90af335de6a: ending tablet copy session eae55bf90c524e8eb9df4c31f40096e4-25df2da40e524e9c95cbe13a484445b5 on tablet 25df2da40e524e9c95cbe13a484445b5 with peer eae55bf90c524e8eb9df4c31f40096e4
I20240502 08:17:18.462270 10429 raft_consensus.cc:1203] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:17:18.489827 11681 tablet_copy_client.cc:523] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:18.493320 11681 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap starting.
I20240502 08:17:18.528728 11681 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:18.529274 11681 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap complete.
I20240502 08:17:18.529572 11681 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.001s
I20240502 08:17:18.530375 11681 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:18.531157 11681 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1f0504c982c42b8b9de29b8830c1e82, State: Initialized, Role: LEARNER
I20240502 08:17:18.531381 11681 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:18.532141 11681 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:18.532986 8933 tablet_copy_service.cc:342] P 71df2128b229478facd8ba1b2683ccca: Request end of tablet copy session d1f0504c982c42b8b9de29b8830c1e82-49af926b61e64eb0ac9c4dcd5b638152 received from {username='slave'} at 127.8.68.197:53343
I20240502 08:17:18.533098 8933 tablet_copy_service.cc:433] P 71df2128b229478facd8ba1b2683ccca: ending tablet copy session d1f0504c982c42b8b9de29b8830c1e82-49af926b61e64eb0ac9c4dcd5b638152 on tablet 49af926b61e64eb0ac9c4dcd5b638152 with peer d1f0504c982c42b8b9de29b8830c1e82
W20240502 08:17:18.644045 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:17:18.888703 10168 raft_consensus.cc:1203] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 08:17:18.901077 11824 raft_consensus.cc:1050] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER 4586652e013946998246ca5489ae7083 to VOTER
I20240502 08:17:18.901793 11824 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:18.904559 10168 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.904951 10299 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.905174 8912 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.905426 11823 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:18.905866 11824 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:18.906271 11826 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:18.910236 11826 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.910933 8912 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.915122 10299 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.915460 10168 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.916896 8508 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 8 to 9, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.926157 8781 consensus_queue.cc:237] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:18.928244 10168 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:17:18.928371 10299 raft_consensus.cc:1261] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:17:18.928860 11824 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:18.929209 11826 consensus_queue.cc:1035] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:18.931610 11722 raft_consensus.cc:1050] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: attempting to promote NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 to VOTER
I20240502 08:17:18.932608 11722 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:18.932741 11826 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.933555 10168 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.936820 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.938514 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9479bf73bb1400c83c247f3654bb79b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:18.938724 10299 raft_consensus.cc:2932] T c9479bf73bb1400c83c247f3654bb79b P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:18.939183 8509 catalog_manager.cc:5484] T c9479bf73bb1400c83c247f3654bb79b P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 9 to 10, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:18.939796 10562 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.940521 11790 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:18.941664 10429 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:18.941792 11790 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 08:17:18.942937 11722 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:18.945854 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet c9479bf73bb1400c83c247f3654bb79b with delete_type TABLET_DATA_TOMBSTONED (TS 71df2128b229478facd8ba1b2683ccca not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49784
I20240502 08:17:18.947636 11839 tablet_replica.cc:330] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: stopping tablet replica
I20240502 08:17:18.948159 11839 raft_consensus.cc:2229] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:18.948606 11839 raft_consensus.cc:2258] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:18.951068 11839 ts_tablet_manager.cc:1875] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:18.955585 11722 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:18.960101 11839 ts_tablet_manager.cc:1888] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:17:18.960407 11839 log.cc:1196] T c9479bf73bb1400c83c247f3654bb79b P 71df2128b229478facd8ba1b2683ccca: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/c9479bf73bb1400c83c247f3654bb79b
I20240502 08:17:18.960351 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:18.961794 8494 catalog_manager.cc:4830] TS 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): tablet c9479bf73bb1400c83c247f3654bb79b (table rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]) successfully deleted
I20240502 08:17:18.962384 10562 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:18.963780 8507 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 8 to 9, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:18.980142 10429 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:19.031508 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46884
I20240502 08:17:19.031875 9041 raft_consensus.cc:595] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:19.045213 9041 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:19.047857 8781 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:19.048099 8912 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:17:19.048135 8979 consensus_peers.cc:485] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a -> Peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925): Couldn't send request to peer 4f6d482dba7d46b89e70150a61cb5cf0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2de1aca9a44a489972aab5decf3620. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:19.048555 11722 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:19.048909 11790 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:19.052423 11790 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:19.053037 8781 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:19.056334 8912 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:19.058862 8507 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
W20240502 08:17:19.097430 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 08:17:19.098879 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 888fe8d0871a4a539b43569e0eda4bef: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:19.100960 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46920
I20240502 08:17:19.101218 9041 raft_consensus.cc:595] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:19.197723 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:17:19.343931 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46946
I20240502 08:17:19.344177 9041 raft_consensus.cc:595] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:19.363444 11835 raft_consensus.cc:979] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: : Instructing follower 4f6d482dba7d46b89e70150a61cb5cf0 to start an election
I20240502 08:17:19.363675 11835 raft_consensus.cc:1067] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Signalling peer 4f6d482dba7d46b89e70150a61cb5cf0 to start an election
I20240502 08:17:19.364856 10562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5"
dest_uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
from {username='slave'} at 127.8.68.196:38709
I20240502 08:17:19.365226 10562 raft_consensus.cc:482] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:19.365448 10562 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:19.365798 11913 raft_consensus.cc:1050] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca: attempting to promote NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 to VOTER
I20240502 08:17:19.366510 11913 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:19.368144 10562 raft_consensus.cc:504] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:19.369943 10168 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.369987 10429 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.370584 11680 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.370715 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:17:19.370872 8782 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.370937 8781 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:19.371661 11680 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.372148 11680 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.373342 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:17:19.373557 9041 raft_consensus.cc:3032] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:19.373665 9041 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:17:19.373911 8781 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
I20240502 08:17:19.374006 9041 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:19.374518 9041 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:19.376888 10562 leader_election.cc:288] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [CANDIDATE]: Term 2 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071)
I20240502 08:17:19.377794 9041 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
I20240502 08:17:19.378314 10494 leader_election.cc:302] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 4f6d482dba7d46b89e70150a61cb5cf0, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:17:19.378762 11846 raft_consensus.cc:2781] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:19.379288 11846 raft_consensus.cc:686] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Becoming Leader. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Running, Role: LEADER
I20240502 08:17:19.380035 11846 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:19.381085 11680 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.383448 10168 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.384846 8509 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: term changed from 1 to 2, leader changed from 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) to 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200). New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:19.389937 8507 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 reported cstate change: config changed from index 9 to 10, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.396598 10429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25df2da40e524e9c95cbe13a484445b5" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "eae55bf90c524e8eb9df4c31f40096e4"
I20240502 08:17:19.396975 10429 raft_consensus.cc:3037] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:19.397490 8782 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.400205 10429 raft_consensus.cc:2454] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
I20240502 08:17:19.401217 8912 consensus_queue.cc:237] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:19.401916 10428 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.404247 10168 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:19.405050 11913 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:19.407222 10427 raft_consensus.cc:1261] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:19.409000 11915 consensus_queue.cc:1035] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:19.409204 11914 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.410465 10427 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.412480 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49af926b61e64eb0ac9c4dcd5b638152 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:19.412469 10168 raft_consensus.cc:2932] T 49af926b61e64eb0ac9c4dcd5b638152 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:19.414458 8507 catalog_manager.cc:5484] T 49af926b61e64eb0ac9c4dcd5b638152 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 10 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:19.418928 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet 49af926b61e64eb0ac9c4dcd5b638152 with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:19.420454 11923 tablet_replica.cc:330] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:19.420877 11923 raft_consensus.cc:2229] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.421175 11923 raft_consensus.cc:2258] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.422801 11923 ts_tablet_manager.cc:1875] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:19.427986 11923 ts_tablet_manager.cc:1888] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:17:19.428159 11923 log.cc:1196] T 49af926b61e64eb0ac9c4dcd5b638152 P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/49af926b61e64eb0ac9c4dcd5b638152
I20240502 08:17:19.429155 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet 49af926b61e64eb0ac9c4dcd5b638152 (table rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]) successfully deleted
I20240502 08:17:19.631022 11950 ts_tablet_manager.cc:902] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Initiating tablet copy from peer 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:17:19.632525 11950 tablet_copy_client.cc:308] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:35907
I20240502 08:17:19.633388 9063 tablet_copy_service.cc:140] P 78c8d87aa5e44674959cd90af335de6a: Received BeginTabletCopySession request for tablet 4e2de1aca9a44a489972aab5decf3620 from peer 4f6d482dba7d46b89e70150a61cb5cf0 ({username='slave'} at 127.8.68.200:37429)
I20240502 08:17:19.633545 9063 tablet_copy_service.cc:161] P 78c8d87aa5e44674959cd90af335de6a: Beginning new tablet copy session on tablet 4e2de1aca9a44a489972aab5decf3620 from peer 4f6d482dba7d46b89e70150a61cb5cf0 at {username='slave'} at 127.8.68.200:37429: session id = 4f6d482dba7d46b89e70150a61cb5cf0-4e2de1aca9a44a489972aab5decf3620
I20240502 08:17:19.635673 9063 tablet_copy_source_session.cc:205] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:19.637097 11950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2de1aca9a44a489972aab5decf3620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.642248 11950 tablet_copy_client.cc:783] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:19.642558 11950 tablet_copy_client.cc:647] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:19.644479 11954 tablet_copy_client.cc:1002] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:19.806780 10427 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.807639 11846 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.906988 8782 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.907691 11846 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.913028 9041 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:19.913663 11846 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:19.925276 10562 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:19.927170 8782 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:19.927235 10427 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:19.927845 11967 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:19.928095 11846 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:19.930544 11970 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:19.931155 8782 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:19.931218 10427 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:19.934609 8497 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25df2da40e524e9c95cbe13a484445b5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:19.936762 8503 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 reported cstate change: config changed from index 9 to 11, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:19.941399 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet 25df2da40e524e9c95cbe13a484445b5 with delete_type TABLET_DATA_TOMBSTONED (TS 78c8d87aa5e44674959cd90af335de6a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52836
I20240502 08:17:19.942281 11976 tablet_replica.cc:330] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: stopping tablet replica
I20240502 08:17:19.942679 11976 raft_consensus.cc:2229] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:19.942992 11976 raft_consensus.cc:2258] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:19.944540 11976 ts_tablet_manager.cc:1875] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:19.949682 11976 ts_tablet_manager.cc:1888] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:17:19.949838 11976 log.cc:1196] T 25df2da40e524e9c95cbe13a484445b5 P 78c8d87aa5e44674959cd90af335de6a: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/25df2da40e524e9c95cbe13a484445b5
I20240502 08:17:19.950685 8494 catalog_manager.cc:4830] TS 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907): tablet 25df2da40e524e9c95cbe13a484445b5 (table rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]) successfully deleted
W20240502 08:17:20.004274 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:17:20.067873 8782 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:20.071030 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:20.071012 8912 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:20.071830 11997 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:20.072260 11996 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:20.075421 11826 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:20.076061 9041 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:20.080188 8912 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
W20240502 08:17:20.081112 8718 consensus_peers.cc:485] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 531f3f63a6c0426987a56f9f8d5e0c56. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:20.082386 8507 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index -1 to 4, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:20.107010 8782 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:20.110297 10562 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:17:20.110507 8716 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 888fe8d0871a4a539b43569e0eda4bef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:20.110841 9041 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:20.111388 11826 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:20.111758 11997 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:20.115021 11997 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:20.115799 10562 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:20.120293 9041 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:20.122485 8507 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
W20240502 08:17:20.542986 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:17:20.603075 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:20.611291 12058 ts_tablet_manager.cc:902] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:20.614742 12058 tablet_copy_client.cc:308] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:20.615608 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet 888fe8d0871a4a539b43569e0eda4bef from peer d1f0504c982c42b8b9de29b8830c1e82 ({username='slave'} at 127.8.68.197:59271)
I20240502 08:17:20.615808 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet 888fe8d0871a4a539b43569e0eda4bef from peer d1f0504c982c42b8b9de29b8830c1e82 at {username='slave'} at 127.8.68.197:59271: session id = d1f0504c982c42b8b9de29b8830c1e82-888fe8d0871a4a539b43569e0eda4bef
I20240502 08:17:20.618258 8802 tablet_copy_source_session.cc:205] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:20.619506 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.624358 12058 tablet_copy_client.cc:783] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:20.624689 12058 tablet_copy_client.cc:647] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:20.626331 12062 tablet_copy_client.cc:1002] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:20.687893 12063 ts_tablet_manager.cc:902] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:20.689098 12063 tablet_copy_client.cc:308] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:20.696040 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet 531f3f63a6c0426987a56f9f8d5e0c56 from peer eae55bf90c524e8eb9df4c31f40096e4 ({username='slave'} at 127.8.68.199:39257)
I20240502 08:17:20.696250 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet 531f3f63a6c0426987a56f9f8d5e0c56 from peer eae55bf90c524e8eb9df4c31f40096e4 at {username='slave'} at 127.8.68.199:39257: session id = eae55bf90c524e8eb9df4c31f40096e4-531f3f63a6c0426987a56f9f8d5e0c56
I20240502 08:17:20.698277 8802 tablet_copy_source_session.cc:205] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:20.699826 12063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531f3f63a6c0426987a56f9f8d5e0c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.705459 12063 tablet_copy_client.cc:783] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:20.705802 12063 tablet_copy_client.cc:647] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:20.707650 12067 tablet_copy_client.cc:1002] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:20.781736 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:17:20.805187 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:17:21.144966 11950 tablet_copy_client.cc:523] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:21.147747 11950 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap starting.
I20240502 08:17:21.188766 11950 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:21.189275 11950 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap complete.
I20240502 08:17:21.189574 11950 ts_tablet_manager.cc:1367] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent bootstrapping tablet: real 0.042s user 0.037s sys 0.001s
I20240502 08:17:21.190390 11950 raft_consensus.cc:348] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:21.190658 11950 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Initialized, Role: LEARNER
I20240502 08:17:21.191277 11950 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:21.191900 11950 ts_tablet_manager.cc:1398] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:17:21.192672 9063 tablet_copy_service.cc:342] P 78c8d87aa5e44674959cd90af335de6a: Request end of tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-4e2de1aca9a44a489972aab5decf3620 received from {username='slave'} at 127.8.68.200:37429
I20240502 08:17:21.192816 9063 tablet_copy_service.cc:433] P 78c8d87aa5e44674959cd90af335de6a: ending tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-4e2de1aca9a44a489972aab5decf3620 on tablet 4e2de1aca9a44a489972aab5decf3620 with peer 4f6d482dba7d46b89e70150a61cb5cf0
W20240502 08:17:21.194247 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:17:21.382819 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:17:21.500865 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:17:21.644037 10562 raft_consensus.cc:1203] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 08:17:21.680738 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:17:21.735481 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:17:22.116005 12165 raft_consensus.cc:1050] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: attempting to promote NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 to VOTER
I20240502 08:17:22.116994 12165 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:22.119742 10562 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:22.120085 8781 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:22.120108 8912 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:22.120473 12164 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:22.120940 12167 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:22.121362 12165 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:22.125051 12167 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.125983 8781 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.126017 10562 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.126725 12058 tablet_copy_client.cc:523] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:22.130640 12058 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap starting.
I20240502 08:17:22.132077 8509 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.139430 8912 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.144170 9041 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:22.146246 8782 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:22.147234 10562 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:22.148046 12167 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:22.148366 12164 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:22.151178 12164 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.151981 8782 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.155951 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e2de1aca9a44a489972aab5decf3620 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:22.156054 10562 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:22.159977 8509 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:22.164508 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2de1aca9a44a489972aab5decf3620 with delete_type TABLET_DATA_TOMBSTONED (TS 71df2128b229478facd8ba1b2683ccca not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49784
I20240502 08:17:22.165256 12177 tablet_replica.cc:330] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: stopping tablet replica
I20240502 08:17:22.165534 12177 raft_consensus.cc:2229] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:22.165879 12177 raft_consensus.cc:2258] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:22.166713 12177 ts_tablet_manager.cc:1875] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:22.171128 12177 ts_tablet_manager.cc:1888] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:22.171284 12177 log.cc:1196] T 4e2de1aca9a44a489972aab5decf3620 P 71df2128b229478facd8ba1b2683ccca: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e2de1aca9a44a489972aab5decf3620
I20240502 08:17:22.172114 8494 catalog_manager.cc:4830] TS 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): tablet 4e2de1aca9a44a489972aab5decf3620 (table rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]) successfully deleted
I20240502 08:17:22.177640 12058 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:22.178069 12058 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap complete.
I20240502 08:17:22.178350 12058 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Time spent bootstrapping tablet: real 0.048s user 0.034s sys 0.009s
I20240502 08:17:22.178992 12058 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:22.179210 12058 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f0504c982c42b8b9de29b8830c1e82, State: Initialized, Role: LEARNER
I20240502 08:17:22.179448 12058 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:22.180298 12058 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240502 08:17:22.181705 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session d1f0504c982c42b8b9de29b8830c1e82-888fe8d0871a4a539b43569e0eda4bef received from {username='slave'} at 127.8.68.197:59271
I20240502 08:17:22.181883 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session d1f0504c982c42b8b9de29b8830c1e82-888fe8d0871a4a539b43569e0eda4bef on tablet 888fe8d0871a4a539b43569e0eda4bef with peer d1f0504c982c42b8b9de29b8830c1e82
I20240502 08:17:22.208117 12063 tablet_copy_client.cc:523] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:22.210877 12063 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap starting.
I20240502 08:17:22.245745 12063 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:22.246270 12063 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap complete.
I20240502 08:17:22.246594 12063 ts_tablet_manager.cc:1367] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20240502 08:17:22.247545 12063 raft_consensus.cc:348] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:22.247774 12063 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eae55bf90c524e8eb9df4c31f40096e4, State: Initialized, Role: LEARNER
I20240502 08:17:22.247972 12063 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:22.248658 12063 ts_tablet_manager.cc:1398] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:22.249486 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session eae55bf90c524e8eb9df4c31f40096e4-531f3f63a6c0426987a56f9f8d5e0c56 received from {username='slave'} at 127.8.68.199:39257
I20240502 08:17:22.249670 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session eae55bf90c524e8eb9df4c31f40096e4-531f3f63a6c0426987a56f9f8d5e0c56 on tablet 531f3f63a6c0426987a56f9f8d5e0c56 with peer eae55bf90c524e8eb9df4c31f40096e4
W20240502 08:17:22.257290 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:17:22.277882 8782 consensus_queue.cc:237] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
W20240502 08:17:22.280599 8715 consensus_peers.cc:485] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 -> Peer 4586652e013946998246ca5489ae7083 (127.8.68.198:36779): Couldn't send request to peer 4586652e013946998246ca5489ae7083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a2f85e56bc14589b9535fd1efe64e9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:22.280895 9041 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:22.280911 8912 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:22.281469 12203 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:22.281834 12102 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:22.284832 12102 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:22.285668 9041 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:22.290254 8503 catalog_manager.cc:5484] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 4, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:22.292286 8912 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
W20240502 08:17:22.470717 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:22.666329 10427 raft_consensus.cc:1203] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:22.688906 10168 raft_consensus.cc:1203] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:17:22.740582 12239 ts_tablet_manager.cc:902] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:22.741775 12239 tablet_copy_client.cc:308] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:22.742481 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet 7a2f85e56bc14589b9535fd1efe64e9d from peer 4586652e013946998246ca5489ae7083 ({username='slave'} at 127.8.68.198:45733)
I20240502 08:17:22.742621 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet 7a2f85e56bc14589b9535fd1efe64e9d from peer 4586652e013946998246ca5489ae7083 at {username='slave'} at 127.8.68.198:45733: session id = 4586652e013946998246ca5489ae7083-7a2f85e56bc14589b9535fd1efe64e9d
I20240502 08:17:22.744632 8802 tablet_copy_source_session.cc:205] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:22.745774 12239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f85e56bc14589b9535fd1efe64e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:22.750253 12239 tablet_copy_client.cc:783] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:22.750504 12239 tablet_copy_client.cc:647] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:22.752238 12243 tablet_copy_client.cc:1002] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:23.018548 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:17:23.136325 12275 raft_consensus.cc:1050] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 to VOTER
I20240502 08:17:23.137243 12275 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:23.140545 9041 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:23.140738 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:23.140810 10562 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:23.141383 12277 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:23.141862 12275 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:23.142166 12102 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:23.148741 12102 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.149699 10562 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.149605 10168 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.154002 9041 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.155285 8503 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.163091 8782 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:23.164839 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:17:23.165028 10562 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:17:23.165485 12203 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:23.165978 12275 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:23.168126 12203 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.168900 10168 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.169548 10561 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.172535 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 888fe8d0871a4a539b43569e0eda4bef with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:23.174036 8508 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 reported cstate change: config changed from index 9 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:23.178145 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet 888fe8d0871a4a539b43569e0eda4bef with delete_type TABLET_DATA_TOMBSTONED (TS 78c8d87aa5e44674959cd90af335de6a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52836
I20240502 08:17:23.179113 12284 tablet_replica.cc:330] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: stopping tablet replica
I20240502 08:17:23.179353 12284 raft_consensus.cc:2229] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:23.179773 12284 raft_consensus.cc:2258] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:23.180944 12284 ts_tablet_manager.cc:1875] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:23.187664 12284 ts_tablet_manager.cc:1888] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:17:23.187808 12284 log.cc:1196] T 888fe8d0871a4a539b43569e0eda4bef P 78c8d87aa5e44674959cd90af335de6a: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/888fe8d0871a4a539b43569e0eda4bef
I20240502 08:17:23.188606 8494 catalog_manager.cc:4830] TS 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907): tablet 888fe8d0871a4a539b43569e0eda4bef (table rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]) successfully deleted
W20240502 08:17:23.252480 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:17:23.284855 12277 raft_consensus.cc:1050] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 to VOTER
I20240502 08:17:23.285616 12277 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:23.287890 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:23.287933 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:23.288051 8911 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:23.288543 12277 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:23.288869 12275 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:23.289147 12102 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:23.293888 12275 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:23.294308 10427 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:23.297435 8911 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:23.298491 9041 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:23.301445 8507 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 4 to 5, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:23.324543 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:17:23.497201 8782 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35396
I20240502 08:17:23.497470 8782 raft_consensus.cc:595] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:23.694749 12102 raft_consensus.cc:979] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: : Instructing follower 71df2128b229478facd8ba1b2683ccca to start an election
I20240502 08:17:23.694980 12102 raft_consensus.cc:1067] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Signalling peer 71df2128b229478facd8ba1b2683ccca to start an election
I20240502 08:17:23.695844 8911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "71df2128b229478facd8ba1b2683ccca"
from {username='slave'} at 127.8.68.194:56755
I20240502 08:17:23.696058 8911 raft_consensus.cc:482] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:23.696157 8911 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.698022 8911 raft_consensus.cc:504] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:23.699348 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:17:23.699383 8911 leader_election.cc:288] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071)
I20240502 08:17:23.699422 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:17:23.699540 8782 raft_consensus.cc:3032] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:23.699631 9041 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.699636 8782 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:17:23.699947 10427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "71df2128b229478facd8ba1b2683ccca" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "eae55bf90c524e8eb9df4c31f40096e4"
I20240502 08:17:23.699980 8782 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:23.700157 10427 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.700410 8782 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.702256 8782 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 2.
I20240502 08:17:23.702378 9041 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 2.
I20240502 08:17:23.702945 10427 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71df2128b229478facd8ba1b2683ccca in term 2.
I20240502 08:17:23.703416 8846 leader_election.cc:302] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:17:23.703760 12330 raft_consensus.cc:2781] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:23.703913 12330 raft_consensus.cc:686] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Becoming Leader. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:17:23.704258 12330 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:23.707697 8508 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca reported cstate change: term changed from 1 to 2, leader changed from 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) to 71df2128b229478facd8ba1b2683ccca (127.8.68.195). New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:23.754616 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:17:23.838692 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:17:23.879058 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c9479bf73bb1400c83c247f3654bb79b: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:24.172264 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:24.173209 12330 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:24.218761 8782 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:24.219434 12330 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:24.224197 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:24.224857 12369 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:24.252740 12239 tablet_copy_client.cc:523] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:24.255733 12239 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Bootstrap starting.
I20240502 08:17:24.288285 12239 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:24.288717 12239 tablet_bootstrap.cc:492] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Bootstrap complete.
I20240502 08:17:24.288985 12239 ts_tablet_manager.cc:1367] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.014s
I20240502 08:17:24.289726 12239 raft_consensus.cc:348] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:24.289932 12239 raft_consensus.cc:729] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4586652e013946998246ca5489ae7083, State: Initialized, Role: LEARNER
I20240502 08:17:24.290132 12239 consensus_queue.cc:260] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:24.290818 12239 ts_tablet_manager.cc:1398] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:17:24.291673 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session 4586652e013946998246ca5489ae7083-7a2f85e56bc14589b9535fd1efe64e9d received from {username='slave'} at 127.8.68.198:45733
I20240502 08:17:24.291846 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session 4586652e013946998246ca5489ae7083-7a2f85e56bc14589b9535fd1efe64e9d on tablet 7a2f85e56bc14589b9535fd1efe64e9d with peer 4586652e013946998246ca5489ae7083
I20240502 08:17:24.642290 8911 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:24.644233 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:17:24.644373 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:17:24.644856 12397 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:24.645195 12330 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:24.647382 12369 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:24.648224 9041 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:24.650318 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 531f3f63a6c0426987a56f9f8d5e0c56 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:24.650816 10427 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:24.653554 8509 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 5 to 7, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:24.659826 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet 531f3f63a6c0426987a56f9f8d5e0c56 with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:24.661550 12401 tablet_replica.cc:330] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:24.661840 12401 raft_consensus.cc:2229] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:24.662253 12401 raft_consensus.cc:2258] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:24.663134 12401 ts_tablet_manager.cc:1875] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:24.668160 12401 ts_tablet_manager.cc:1888] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:24.668309 12401 log.cc:1196] T 531f3f63a6c0426987a56f9f8d5e0c56 P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/531f3f63a6c0426987a56f9f8d5e0c56
I20240502 08:17:24.669068 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet 531f3f63a6c0426987a56f9f8d5e0c56 (table rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]) successfully deleted
W20240502 08:17:24.705147 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:17:24.783440 10299 raft_consensus.cc:1203] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:24.854281 8911 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:24.856945 8782 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:24.856945 9041 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:24.857589 12330 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:24.858177 12397 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:24.861040 12330 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:24.861764 8782 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:24.862063 9041 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:24.867115 8508 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index -1 to 4, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
W20240502 08:17:24.873188 8849 consensus_peers.cc:485] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca -> Peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925): Couldn't send request to peer 4f6d482dba7d46b89e70150a61cb5cf0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e29f136434ad4ff1b0c9d4ac4481e216. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:25.214408 12442 raft_consensus.cc:1050] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER 4586652e013946998246ca5489ae7083 to VOTER
I20240502 08:17:25.215365 12442 consensus_queue.cc:237] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:25.217948 10299 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:25.218473 8911 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:25.218622 9041 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:25.219097 12442 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:25.219456 12102 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:25.219772 12441 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:25.223435 12444 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.226792 8911 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.227862 9041 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.232596 8509 catalog_manager.cc:5484] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 4 to 5, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:25.234392 10299 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.241092 8782 consensus_queue.cc:237] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:25.242851 9041 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:25.242913 10299 raft_consensus.cc:1261] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:25.243422 12442 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:25.243738 12441 consensus_queue.cc:1035] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:25.246131 12444 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.246739 9041 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.246692 10299 raft_consensus.cc:2932] T 7a2f85e56bc14589b9535fd1efe64e9d P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.250437 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a2f85e56bc14589b9535fd1efe64e9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:25.252746 8509 catalog_manager.cc:5484] T 7a2f85e56bc14589b9535fd1efe64e9d P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 5 to 6, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:25.256223 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2f85e56bc14589b9535fd1efe64e9d with delete_type TABLET_DATA_TOMBSTONED (TS 71df2128b229478facd8ba1b2683ccca not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49784
I20240502 08:17:25.257030 12448 tablet_replica.cc:330] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: stopping tablet replica
I20240502 08:17:25.257248 12448 raft_consensus.cc:2229] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:25.257678 12448 raft_consensus.cc:2258] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:25.258534 12448 ts_tablet_manager.cc:1875] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:25.263583 12448 ts_tablet_manager.cc:1888] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:25.263703 12448 log.cc:1196] T 7a2f85e56bc14589b9535fd1efe64e9d P 71df2128b229478facd8ba1b2683ccca: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a2f85e56bc14589b9535fd1efe64e9d
I20240502 08:17:25.264482 8494 catalog_manager.cc:4830] TS 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): tablet 7a2f85e56bc14589b9535fd1efe64e9d (table rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]) successfully deleted
W20240502 08:17:25.460134 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:17:25.479423 12465 ts_tablet_manager.cc:902] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Initiating tablet copy from peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:25.480662 12465 tablet_copy_client.cc:308] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:36747
I20240502 08:17:25.486945 8933 tablet_copy_service.cc:140] P 71df2128b229478facd8ba1b2683ccca: Received BeginTabletCopySession request for tablet e29f136434ad4ff1b0c9d4ac4481e216 from peer 4f6d482dba7d46b89e70150a61cb5cf0 ({username='slave'} at 127.8.68.200:39407)
I20240502 08:17:25.487692 8933 tablet_copy_service.cc:161] P 71df2128b229478facd8ba1b2683ccca: Beginning new tablet copy session on tablet e29f136434ad4ff1b0c9d4ac4481e216 from peer 4f6d482dba7d46b89e70150a61cb5cf0 at {username='slave'} at 127.8.68.200:39407: session id = 4f6d482dba7d46b89e70150a61cb5cf0-e29f136434ad4ff1b0c9d4ac4481e216
I20240502 08:17:25.489677 8933 tablet_copy_source_session.cc:205] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:25.490967 12465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29f136434ad4ff1b0c9d4ac4481e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.495676 12465 tablet_copy_client.cc:783] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:25.495940 12465 tablet_copy_client.cc:647] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:25.497643 12469 tablet_copy_client.cc:1002] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:25.612062 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 08:17:25.613442 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25df2da40e524e9c95cbe13a484445b5: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:25.730963 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:26.056417 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:26.066896 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:26.184419 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:17:26.617426 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:26.813419 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:26.830367 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:17:26.998059 12465 tablet_copy_client.cc:523] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.000794 12465 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap starting.
I20240502 08:17:27.031446 12465 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:27.031890 12465 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap complete.
I20240502 08:17:27.032146 12465 ts_tablet_manager.cc:1367] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.008s
I20240502 08:17:27.032827 12465 raft_consensus.cc:348] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:27.033042 12465 raft_consensus.cc:729] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Initialized, Role: LEARNER
I20240502 08:17:27.033248 12465 consensus_queue.cc:260] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:27.033761 12465 ts_tablet_manager.cc:1398] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:27.034721 8933 tablet_copy_service.cc:342] P 71df2128b229478facd8ba1b2683ccca: Request end of tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-e29f136434ad4ff1b0c9d4ac4481e216 received from {username='slave'} at 127.8.68.200:39407
I20240502 08:17:27.034879 8933 tablet_copy_service.cc:433] P 71df2128b229478facd8ba1b2683ccca: ending tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-e29f136434ad4ff1b0c9d4ac4481e216 on tablet e29f136434ad4ff1b0c9d4ac4481e216 with peer 4f6d482dba7d46b89e70150a61cb5cf0
W20240502 08:17:27.066783 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:17:27.480034 10561 raft_consensus.cc:1203] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:27.889983 12536 raft_consensus.cc:1050] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: attempting to promote NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 to VOTER
I20240502 08:17:27.891065 12536 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
W20240502 08:17:27.891821 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:17:27.894685 9041 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:27.894745 10561 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:27.895298 8781 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:27.895437 12536 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:27.895884 12488 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:27.896225 12537 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:27.899941 12488 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.901130 9041 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.902253 8781 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.904978 10561 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.906801 8509 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 4 to 5, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:27.913888 8911 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:27.915700 9041 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:27.915849 10561 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:27.916355 12488 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:27.916797 12536 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:27.918648 12488 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.919521 9041 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.919785 10561 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.922600 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e29f136434ad4ff1b0c9d4ac4481e216 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:27.925032 8507 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 5 to 6, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:27.928802 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet e29f136434ad4ff1b0c9d4ac4481e216 with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:27.929456 12551 tablet_replica.cc:330] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:27.929662 12551 raft_consensus.cc:2229] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:27.929976 12551 raft_consensus.cc:2258] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:27.930905 12551 ts_tablet_manager.cc:1875] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:27.935814 12551 ts_tablet_manager.cc:1888] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:27.935985 12551 log.cc:1196] T e29f136434ad4ff1b0c9d4ac4481e216 P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/e29f136434ad4ff1b0c9d4ac4481e216
I20240502 08:17:27.937331 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet e29f136434ad4ff1b0c9d4ac4481e216 (table rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]) successfully deleted
I20240502 08:17:28.130932 8781 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:28.133684 9041 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:28.133694 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:17:28.134317 8718 consensus_peers.cc:485] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925): Couldn't send request to peer 4f6d482dba7d46b89e70150a61cb5cf0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8896dce80734a7b85155855d6cb3303. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:28.134474 12565 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:28.134866 12564 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:28.137640 12565 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:28.138437 10299 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:28.138579 9041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } } }
I20240502 08:17:28.142853 8508 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 7 to 8, NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:28.414872 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:17:28.433812 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:17:28.537437 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:17:28.703019 12584 ts_tablet_manager.cc:902] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:28.705029 12584 tablet_copy_client.cc:308] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:28.705817 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet e8896dce80734a7b85155855d6cb3303 from peer 4f6d482dba7d46b89e70150a61cb5cf0 ({username='slave'} at 127.8.68.200:34981)
I20240502 08:17:28.706003 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet e8896dce80734a7b85155855d6cb3303 from peer 4f6d482dba7d46b89e70150a61cb5cf0 at {username='slave'} at 127.8.68.200:34981: session id = 4f6d482dba7d46b89e70150a61cb5cf0-e8896dce80734a7b85155855d6cb3303
W20240502 08:17:28.707121 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:17:28.708449 8802 tablet_copy_source_session.cc:205] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:28.709687 12584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8896dce80734a7b85155855d6cb3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:28.714073 12584 tablet_copy_client.cc:783] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:28.714385 12584 tablet_copy_client.cc:647] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:28.715977 12588 tablet_copy_client.cc:1002] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:28.810856 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:29.351086 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:17:30.113641 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:30.216463 12584 tablet_copy_client.cc:523] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:30.219201 12584 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap starting.
I20240502 08:17:30.252897 12584 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:30.253427 12584 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Bootstrap complete.
I20240502 08:17:30.253705 12584 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.005s
I20240502 08:17:30.254384 12584 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:30.254611 12584 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Initialized, Role: LEARNER
I20240502 08:17:30.254796 12584 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:30.255439 12584 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:30.255508 10561 raft_consensus.cc:1203] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:17:30.256301 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-e8896dce80734a7b85155855d6cb3303 received from {username='slave'} at 127.8.68.200:34981
I20240502 08:17:30.256446 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session 4f6d482dba7d46b89e70150a61cb5cf0-e8896dce80734a7b85155855d6cb3303 on tablet e8896dce80734a7b85155855d6cb3303 with peer 4f6d482dba7d46b89e70150a61cb5cf0
I20240502 08:17:30.715363 12565 consensus_queue.cc:579] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.129s)
I20240502 08:17:30.726388 12628 consensus_queue.cc:579] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.054s)
I20240502 08:17:30.818434 12644 raft_consensus.cc:1050] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER 4f6d482dba7d46b89e70150a61cb5cf0 to VOTER
I20240502 08:17:30.819128 12644 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:30.822593 9041 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:30.822705 10561 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:30.822815 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:30.823297 12648 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:30.823700 12644 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:30.824010 12645 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:30.839080 12565 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
W20240502 08:17:30.840504 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:30.844128 10299 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:30.844763 9041 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:30.845175 10561 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:30.848194 8507 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 8 to 9, 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:30.849660 12167 consensus_queue.cc:579] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.138s)
I20240502 08:17:30.910511 8781 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59762
I20240502 08:17:30.910724 8781 raft_consensus.cc:595] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:31.048884 12656 consensus_queue.cc:579] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.470s)
W20240502 08:17:31.050711 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:31.130379 8781 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59778
I20240502 08:17:31.130581 8781 raft_consensus.cc:595] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:31.134045 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:31.350832 8781 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59802
I20240502 08:17:31.351038 8781 raft_consensus.cc:595] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:31.378070 12644 raft_consensus.cc:979] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: : Instructing follower 78c8d87aa5e44674959cd90af335de6a to start an election
I20240502 08:17:31.378260 12644 raft_consensus.cc:1067] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Signalling peer 78c8d87aa5e44674959cd90af335de6a to start an election
I20240502 08:17:31.379079 9041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
from {username='slave'} at 127.8.68.194:57701
I20240502 08:17:31.379352 9041 raft_consensus.cc:482] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:31.379493 9041 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:31.381489 9041 raft_consensus.cc:504] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:31.382407 9041 leader_election.cc:288] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 2 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), 4586652e013946998246ca5489ae7083 (127.8.68.198:36779), 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925)
I20240502 08:17:31.383350 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:17:31.383414 10561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
I20240502 08:17:31.383596 8781 raft_consensus.cc:3032] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:31.383705 8781 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:17:31.383721 10561 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:31.384029 8781 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:31.384505 8781 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:31.386761 10561 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 2.
I20240502 08:17:31.387395 8781 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 2.
I20240502 08:17:31.387883 8977 leader_election.cc:302] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 4f6d482dba7d46b89e70150a61cb5cf0, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:17:31.388180 12656 raft_consensus.cc:2781] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:31.388329 12656 raft_consensus.cc:686] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 2 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:17:31.388708 12656 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:31.390415 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8896dce80734a7b85155855d6cb3303" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "4586652e013946998246ca5489ae7083"
I20240502 08:17:31.390779 10299 raft_consensus.cc:3037] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:31.393033 10299 raft_consensus.cc:2454] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 2.
I20240502 08:17:31.394250 8503 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 1 to 2, leader changed from 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 2 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:31.398494 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:31.451138 12646 consensus_queue.cc:579] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.524s)
I20240502 08:17:31.837464 10561 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:31.838608 12656 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:17:31.843973 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:17:31.917959 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:17:31.950557 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:31.951210 12656 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:31.957494 8781 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:31.961503 9041 consensus_queue.cc:237] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } }
I20240502 08:17:31.963495 10299 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:31.964080 12711 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:31.966519 12714 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:31.967339 10299 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
I20240502 08:17:31.970038 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8896dce80734a7b85155855d6cb3303 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:31.971271 8509 catalog_manager.cc:5484] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 9 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:31.976462 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet e8896dce80734a7b85155855d6cb3303 with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:31.977279 12719 tablet_replica.cc:330] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:31.977491 12719 raft_consensus.cc:2229] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:31.977696 12719 raft_consensus.cc:2258] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:31.978513 12719 ts_tablet_manager.cc:1875] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:31.979629 10562 raft_consensus.cc:1261] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:17:31.980623 12711 consensus_queue.cc:1035] T e8896dce80734a7b85155855d6cb3303 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:31.984212 12719 ts_tablet_manager.cc:1888] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:17:31.984330 12719 log.cc:1196] T e8896dce80734a7b85155855d6cb3303 P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8896dce80734a7b85155855d6cb3303
I20240502 08:17:31.985107 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet e8896dce80734a7b85155855d6cb3303 (table rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]) successfully deleted
I20240502 08:17:31.988056 10561 raft_consensus.cc:2932] T e8896dce80734a7b85155855d6cb3303 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } }
W20240502 08:17:31.993250 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:32.100574 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:17:32.116345 10542 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:17:32.120110 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 08:17:32.126456 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c9479bf73bb1400c83c247f3654bb79b: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:32.130800 8760 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:32.146821 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:32.156503 8893 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:32.156903 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:32.210228 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:32.210695 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:32.291838 8911 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
W20240502 08:17:32.294338 8849 consensus_peers.cc:485] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e29f136434ad4ff1b0c9d4ac4481e216. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:32.294334 10561 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:32.294378 9041 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:32.294919 12696 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:32.295215 12770 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:32.298513 12770 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:32.298914 9041 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:32.299201 10561 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:32.304219 8503 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 6 to 7, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:32.318300 8911 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:32.321369 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 08:17:32.321453 8847 consensus_peers.cc:485] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 531f3f63a6c0426987a56f9f8d5e0c56. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:32.321893 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:17:32.321975 12773 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:32.322527 12770 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:32.325762 12696 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:32.326550 9041 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:32.329823 10427 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:32.335352 8509 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 7 to 8, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:32.565995 12644 consensus_queue.cc:579] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Leader has been unable to successfully communicate with peer bdf1a2f21da04697b58f19f331492b84 for more than 30 seconds (30.394s)
W20240502 08:17:32.721549 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:17:32.888012 12818 ts_tablet_manager.cc:902] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Initiating tablet copy from peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:32.889521 12818 tablet_copy_client.cc:308] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:36747
I20240502 08:17:32.890254 8933 tablet_copy_service.cc:140] P 71df2128b229478facd8ba1b2683ccca: Received BeginTabletCopySession request for tablet e29f136434ad4ff1b0c9d4ac4481e216 from peer eae55bf90c524e8eb9df4c31f40096e4 ({username='slave'} at 127.8.68.199:33243)
I20240502 08:17:32.890429 8933 tablet_copy_service.cc:161] P 71df2128b229478facd8ba1b2683ccca: Beginning new tablet copy session on tablet e29f136434ad4ff1b0c9d4ac4481e216 from peer eae55bf90c524e8eb9df4c31f40096e4 at {username='slave'} at 127.8.68.199:33243: session id = eae55bf90c524e8eb9df4c31f40096e4-e29f136434ad4ff1b0c9d4ac4481e216
I20240502 08:17:32.892524 8933 tablet_copy_source_session.cc:205] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:32.893740 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29f136434ad4ff1b0c9d4ac4481e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.898411 12818 tablet_copy_client.cc:783] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:32.898708 12818 tablet_copy_client.cc:647] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:32.900626 12822 tablet_copy_client.cc:1002] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:32.933912 12824 ts_tablet_manager.cc:902] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Initiating tablet copy from peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747)
I20240502 08:17:32.935258 12824 tablet_copy_client.cc:308] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:36747
I20240502 08:17:32.935974 8933 tablet_copy_service.cc:140] P 71df2128b229478facd8ba1b2683ccca: Received BeginTabletCopySession request for tablet 531f3f63a6c0426987a56f9f8d5e0c56 from peer d1f0504c982c42b8b9de29b8830c1e82 ({username='slave'} at 127.8.68.197:53343)
I20240502 08:17:32.936125 8933 tablet_copy_service.cc:161] P 71df2128b229478facd8ba1b2683ccca: Beginning new tablet copy session on tablet 531f3f63a6c0426987a56f9f8d5e0c56 from peer d1f0504c982c42b8b9de29b8830c1e82 at {username='slave'} at 127.8.68.197:53343: session id = d1f0504c982c42b8b9de29b8830c1e82-531f3f63a6c0426987a56f9f8d5e0c56
I20240502 08:17:32.938150 8933 tablet_copy_source_session.cc:205] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:32.939353 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 531f3f63a6c0426987a56f9f8d5e0c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.944043 12824 tablet_copy_client.cc:783] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:32.944355 12824 tablet_copy_client.cc:647] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:32.946015 12828 tablet_copy_client.cc:1002] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:33.435173 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:17:33.516971 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:17:33.729857 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:17:34.161381 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:17:34.401074 12818 tablet_copy_client.cc:523] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:34.403877 12818 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap starting.
I20240502 08:17:34.436767 12818 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:34.437189 12818 tablet_bootstrap.cc:492] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap complete.
I20240502 08:17:34.437445 12818 ts_tablet_manager.cc:1367] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.006s
I20240502 08:17:34.438270 12818 raft_consensus.cc:348] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:34.438668 12818 raft_consensus.cc:729] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eae55bf90c524e8eb9df4c31f40096e4, State: Initialized, Role: LEARNER
I20240502 08:17:34.438892 12818 consensus_queue.cc:260] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:34.440393 12818 ts_tablet_manager.cc:1398] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.002s
I20240502 08:17:34.441162 8933 tablet_copy_service.cc:342] P 71df2128b229478facd8ba1b2683ccca: Request end of tablet copy session eae55bf90c524e8eb9df4c31f40096e4-e29f136434ad4ff1b0c9d4ac4481e216 received from {username='slave'} at 127.8.68.199:33243
I20240502 08:17:34.441318 8933 tablet_copy_service.cc:433] P 71df2128b229478facd8ba1b2683ccca: ending tablet copy session eae55bf90c524e8eb9df4c31f40096e4-e29f136434ad4ff1b0c9d4ac4481e216 on tablet e29f136434ad4ff1b0c9d4ac4481e216 with peer eae55bf90c524e8eb9df4c31f40096e4
I20240502 08:17:34.446437 12824 tablet_copy_client.cc:523] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:34.449471 12824 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap starting.
I20240502 08:17:34.483716 12824 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:34.484210 12824 tablet_bootstrap.cc:492] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap complete.
I20240502 08:17:34.484491 12824 ts_tablet_manager.cc:1367] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Time spent bootstrapping tablet: real 0.035s user 0.035s sys 0.000s
I20240502 08:17:34.485188 12824 raft_consensus.cc:348] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:34.485407 12824 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1f0504c982c42b8b9de29b8830c1e82, State: Initialized, Role: LEARNER
I20240502 08:17:34.485600 12824 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:34.486374 12824 ts_tablet_manager.cc:1398] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:34.487308 8933 tablet_copy_service.cc:342] P 71df2128b229478facd8ba1b2683ccca: Request end of tablet copy session d1f0504c982c42b8b9de29b8830c1e82-531f3f63a6c0426987a56f9f8d5e0c56 received from {username='slave'} at 127.8.68.197:53343
I20240502 08:17:34.487493 8933 tablet_copy_service.cc:433] P 71df2128b229478facd8ba1b2683ccca: ending tablet copy session d1f0504c982c42b8b9de29b8830c1e82-531f3f63a6c0426987a56f9f8d5e0c56 on tablet 531f3f63a6c0426987a56f9f8d5e0c56 with peer d1f0504c982c42b8b9de29b8830c1e82
W20240502 08:17:34.488648 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:17:34.498335 10427 raft_consensus.cc:1203] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:17:34.829967 10168 raft_consensus.cc:1203] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:17:34.981734 12928 raft_consensus.cc:1050] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca: attempting to promote NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 to VOTER
I20240502 08:17:34.982734 12928 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:34.986160 10427 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:34.986218 10561 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:34.986523 9041 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:34.986868 12927 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:34.987313 12928 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:34.987679 12861 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:34.991557 12927 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:34.992482 10561 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:34.996353 8508 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca reported cstate change: config changed from index 7 to 8, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:34.996397 9041 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.003777 10427 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.012655 8911 consensus_queue.cc:237] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:35.014959 10561 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:35.015058 10428 raft_consensus.cc:1261] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:17:35.015831 12927 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:35.016285 12928 consensus_queue.cc:1035] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:35.018710 12928 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 71df2128b229478facd8ba1b2683ccca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.020236 10428 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.021500 10561 raft_consensus.cc:2932] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.028072 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e29f136434ad4ff1b0c9d4ac4481e216 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:35.028043 8503 catalog_manager.cc:5484] T e29f136434ad4ff1b0c9d4ac4481e216 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 8 to 9, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:35.031802 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet e29f136434ad4ff1b0c9d4ac4481e216 with delete_type TABLET_DATA_TOMBSTONED (TS 78c8d87aa5e44674959cd90af335de6a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52836
I20240502 08:17:35.032555 12948 tablet_replica.cc:330] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: stopping tablet replica
I20240502 08:17:35.032868 12948 raft_consensus.cc:2229] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:35.033423 12948 raft_consensus.cc:2258] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:35.034492 12948 ts_tablet_manager.cc:1875] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:35.042114 12948 ts_tablet_manager.cc:1888] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:17:35.042311 12948 log.cc:1196] T e29f136434ad4ff1b0c9d4ac4481e216 P 78c8d87aa5e44674959cd90af335de6a: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/e29f136434ad4ff1b0c9d4ac4481e216
I20240502 08:17:35.043443 8494 catalog_manager.cc:4830] TS 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907): tablet e29f136434ad4ff1b0c9d4ac4481e216 (table rebalance_test_table_3 [id=1d58b4d36e4c439fb20e5fb2d0eb23e1]) successfully deleted
I20240502 08:17:35.054174 8781 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:35.056923 10561 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:17:35.056974 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:17:35.057698 12956 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:35.058426 12957 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 08:17:35.058909 8718 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 888fe8d0871a4a539b43569e0eda4bef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:35.061251 12956 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:35.061995 10168 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:35.062398 10561 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } } }
I20240502 08:17:35.066598 8507 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 10 to 11, NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:35.081948 9041 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:35.084581 10561 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:17:35.084836 8976 consensus_peers.cc:485] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a -> Peer 4586652e013946998246ca5489ae7083 (127.8.68.198:36779): Couldn't send request to peer 4586652e013946998246ca5489ae7083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2de1aca9a44a489972aab5decf3620. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:35.084815 8781 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:17:35.085474 12967 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:35.085842 12892 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:17:35.088989 12967 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:35.089439 8781 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:35.089696 10561 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
I20240502 08:17:35.094475 8503 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 6 to 7, NON_VOTER 4586652e013946998246ca5489ae7083 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } } }
W20240502 08:17:35.124540 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:17:35.342938 12928 raft_consensus.cc:1050] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: attempting to promote NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 to VOTER
I20240502 08:17:35.343899 12928 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:35.346772 10168 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:35.347384 9041 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:35.347566 10428 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 71df2128b229478facd8ba1b2683ccca: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:35.347474 12928 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:35.348394 12861 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:35.349207 12927 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 08:17:35.356240 12861 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:35.358538 10168 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:35.363673 9041 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:35.364558 8508 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 reported cstate change: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "71df2128b229478facd8ba1b2683ccca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:35.364251 10428 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:35.386507 8911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "71df2128b229478facd8ba1b2683ccca"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43528
I20240502 08:17:35.386804 8911 raft_consensus.cc:595] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Received request to transfer leadership
I20240502 08:17:35.459617 13013 ts_tablet_manager.cc:902] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Initiating tablet copy from peer 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207)
I20240502 08:17:35.461128 13013 tablet_copy_client.cc:308] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46207
I20240502 08:17:35.461961 8802 tablet_copy_service.cc:140] P 14ea87db2f6a4497a2a961afecd23a32: Received BeginTabletCopySession request for tablet 888fe8d0871a4a539b43569e0eda4bef from peer eae55bf90c524e8eb9df4c31f40096e4 ({username='slave'} at 127.8.68.199:39257)
I20240502 08:17:35.462158 8802 tablet_copy_service.cc:161] P 14ea87db2f6a4497a2a961afecd23a32: Beginning new tablet copy session on tablet 888fe8d0871a4a539b43569e0eda4bef from peer eae55bf90c524e8eb9df4c31f40096e4 at {username='slave'} at 127.8.68.199:39257: session id = eae55bf90c524e8eb9df4c31f40096e4-888fe8d0871a4a539b43569e0eda4bef
I20240502 08:17:35.464378 8802 tablet_copy_source_session.cc:205] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:35.465875 13013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fe8d0871a4a539b43569e0eda4bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.470494 13013 tablet_copy_client.cc:783] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:35.470777 13013 tablet_copy_client.cc:647] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:35.472879 13017 tablet_copy_client.cc:1002] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:35.501148 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 888fe8d0871a4a539b43569e0eda4bef: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:35.536772 13020 ts_tablet_manager.cc:902] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Initiating tablet copy from peer 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907)
I20240502 08:17:35.538551 13020 tablet_copy_client.cc:308] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:35907
I20240502 08:17:35.544813 9063 tablet_copy_service.cc:140] P 78c8d87aa5e44674959cd90af335de6a: Received BeginTabletCopySession request for tablet 4e2de1aca9a44a489972aab5decf3620 from peer 4586652e013946998246ca5489ae7083 ({username='slave'} at 127.8.68.198:48853)
I20240502 08:17:35.544993 9063 tablet_copy_service.cc:161] P 78c8d87aa5e44674959cd90af335de6a: Beginning new tablet copy session on tablet 4e2de1aca9a44a489972aab5decf3620 from peer 4586652e013946998246ca5489ae7083 at {username='slave'} at 127.8.68.198:48853: session id = 4586652e013946998246ca5489ae7083-4e2de1aca9a44a489972aab5decf3620
I20240502 08:17:35.547179 9063 tablet_copy_source_session.cc:205] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:35.548722 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2de1aca9a44a489972aab5decf3620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.554351 13020 tablet_copy_client.cc:783] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:35.554677 13020 tablet_copy_client.cc:647] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:35.556556 13024 tablet_copy_client.cc:1002] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: tablet copy: Injecting latency into file download: 1500
I20240502 08:17:35.628778 8911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "71df2128b229478facd8ba1b2683ccca"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43546
I20240502 08:17:35.629019 8911 raft_consensus.cc:595] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Received request to transfer leadership
W20240502 08:17:35.791066 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:17:35.868279 8911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "71df2128b229478facd8ba1b2683ccca"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43566
I20240502 08:17:35.868577 8911 raft_consensus.cc:595] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Received request to transfer leadership
I20240502 08:17:35.940891 13066 raft_consensus.cc:979] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: : Instructing follower 78c8d87aa5e44674959cd90af335de6a to start an election
I20240502 08:17:35.941095 13066 raft_consensus.cc:1067] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Signalling peer 78c8d87aa5e44674959cd90af335de6a to start an election
I20240502 08:17:35.941959 9041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
from {username='slave'} at 127.8.68.195:53447
I20240502 08:17:35.942216 9041 raft_consensus.cc:482] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:35.942337 9041 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:17:35.944165 9041 raft_consensus.cc:504] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:35.945031 9041 leader_election.cc:288] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 3 election: Requested vote from peers 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747), eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071), d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561)
I20240502 08:17:35.945457 10428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "eae55bf90c524e8eb9df4c31f40096e4"
I20240502 08:17:35.945432 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "71df2128b229478facd8ba1b2683ccca"
I20240502 08:17:35.945675 10428 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:17:35.945679 8911 raft_consensus.cc:3032] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:17:35.945789 8911 raft_consensus.cc:729] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71df2128b229478facd8ba1b2683ccca, State: Running, Role: LEADER
I20240502 08:17:35.946136 8911 consensus_queue.cc:260] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:35.946583 8911 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:17:35.948588 10428 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 3.
I20240502 08:17:35.949283 8911 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 3.
I20240502 08:17:35.949846 8976 leader_election.cc:302] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71df2128b229478facd8ba1b2683ccca, 78c8d87aa5e44674959cd90af335de6a, eae55bf90c524e8eb9df4c31f40096e4; no voters:
I20240502 08:17:35.950511 12967 raft_consensus.cc:2781] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:17:35.951473 12967 raft_consensus.cc:686] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 3 LEADER]: Becoming Leader. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:17:35.951802 12967 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:35.952405 10168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "531f3f63a6c0426987a56f9f8d5e0c56" candidate_uuid: "78c8d87aa5e44674959cd90af335de6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "d1f0504c982c42b8b9de29b8830c1e82"
I20240502 08:17:35.952695 10168 raft_consensus.cc:3037] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:17:35.955576 10168 raft_consensus.cc:2454] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c8d87aa5e44674959cd90af335de6a in term 3.
I20240502 08:17:35.956216 8507 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: term changed from 2 to 3, leader changed from 71df2128b229478facd8ba1b2683ccca (127.8.68.195) to 78c8d87aa5e44674959cd90af335de6a (127.8.68.196). New cstate: current_term: 3 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:36.069919 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:17:36.183835 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:17:36.456952 10168 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 3 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:17:36.457620 13068 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:36.480333 8911 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 3 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:17:36.481007 12967 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:36.481724 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 3 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:17:36.482460 12967 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:36.504819 9041 consensus_queue.cc:237] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:36.506727 10168 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 3 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:17:36.506886 10427 raft_consensus.cc:1261] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 3 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:17:36.507366 12967 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:36.507678 13068 consensus_queue.cc:1035] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:17:36.510375 13074 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:36.511022 10168 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P d1f0504c982c42b8b9de29b8830c1e82 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:36.511126 10427 raft_consensus.cc:2932] T 531f3f63a6c0426987a56f9f8d5e0c56 P eae55bf90c524e8eb9df4c31f40096e4 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:36.513690 8494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 531f3f63a6c0426987a56f9f8d5e0c56 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:36.515188 8507 catalog_manager.cc:5484] T 531f3f63a6c0426987a56f9f8d5e0c56 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 9 to 11, VOTER 71df2128b229478facd8ba1b2683ccca (127.8.68.195) evicted. New cstate: current_term: 3 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:36.520042 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 531f3f63a6c0426987a56f9f8d5e0c56 with delete_type TABLET_DATA_TOMBSTONED (TS 71df2128b229478facd8ba1b2683ccca not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49784
I20240502 08:17:36.520742 13117 tablet_replica.cc:330] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: stopping tablet replica
I20240502 08:17:36.520987 13117 raft_consensus.cc:2229] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:36.521198 13117 raft_consensus.cc:2258] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:36.522047 13117 ts_tablet_manager.cc:1875] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:36.526428 13117 ts_tablet_manager.cc:1888] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 08:17:36.526552 13117 log.cc:1196] T 531f3f63a6c0426987a56f9f8d5e0c56 P 71df2128b229478facd8ba1b2683ccca: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/531f3f63a6c0426987a56f9f8d5e0c56
I20240502 08:17:36.527293 8494 catalog_manager.cc:4830] TS 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): tablet 531f3f63a6c0426987a56f9f8d5e0c56 (table rebalance_test_table_4 [id=7e07b5bc097845cfb3eaed9a180f4823]) successfully deleted
W20240502 08:17:36.713682 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:17:36.866439 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:17:36.973385 13013 tablet_copy_client.cc:523] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:36.976528 13013 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap starting.
I20240502 08:17:37.013767 13013 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:37.014214 13013 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Bootstrap complete.
I20240502 08:17:37.014487 13013 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.001s
I20240502 08:17:37.015986 13013 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:37.016275 13013 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eae55bf90c524e8eb9df4c31f40096e4, State: Initialized, Role: LEARNER
I20240502 08:17:37.016494 13013 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: true } }
I20240502 08:17:37.017397 13013 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.004s
I20240502 08:17:37.018268 8802 tablet_copy_service.cc:342] P 14ea87db2f6a4497a2a961afecd23a32: Request end of tablet copy session eae55bf90c524e8eb9df4c31f40096e4-888fe8d0871a4a539b43569e0eda4bef received from {username='slave'} at 127.8.68.199:39257
I20240502 08:17:37.018437 8802 tablet_copy_service.cc:433] P 14ea87db2f6a4497a2a961afecd23a32: ending tablet copy session eae55bf90c524e8eb9df4c31f40096e4-888fe8d0871a4a539b43569e0eda4bef on tablet 888fe8d0871a4a539b43569e0eda4bef with peer eae55bf90c524e8eb9df4c31f40096e4
W20240502 08:17:37.020152 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:17:37.057057 13020 tablet_copy_client.cc:523] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:37.059929 13020 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Bootstrap starting.
W20240502 08:17:37.066727 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:17:37.092586 13020 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:37.093003 13020 tablet_bootstrap.cc:492] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Bootstrap complete.
I20240502 08:17:37.093322 13020 ts_tablet_manager.cc:1367] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240502 08:17:37.093966 13020 raft_consensus.cc:348] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:37.094169 13020 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4586652e013946998246ca5489ae7083, State: Initialized, Role: LEARNER
I20240502 08:17:37.094357 13020 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: true } }
I20240502 08:17:37.095070 13020 ts_tablet_manager.cc:1398] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:17:37.095976 9063 tablet_copy_service.cc:342] P 78c8d87aa5e44674959cd90af335de6a: Request end of tablet copy session 4586652e013946998246ca5489ae7083-4e2de1aca9a44a489972aab5decf3620 received from {username='slave'} at 127.8.68.198:48853
I20240502 08:17:37.096098 9063 tablet_copy_service.cc:433] P 78c8d87aa5e44674959cd90af335de6a: ending tablet copy session 4586652e013946998246ca5489ae7083-4e2de1aca9a44a489972aab5decf3620 on tablet 4e2de1aca9a44a489972aab5decf3620 with peer 4586652e013946998246ca5489ae7083
W20240502 08:17:37.186034 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:37.247475 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:17:37.431600 10427 raft_consensus.cc:1203] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 08:17:37.622311 10299 raft_consensus.cc:1203] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:17:37.929173 13213 raft_consensus.cc:1050] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: attempting to promote NON_VOTER eae55bf90c524e8eb9df4c31f40096e4 to VOTER
I20240502 08:17:37.929877 13213 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:37.932929 10561 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:17:37.932929 10427 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 LEARNER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:17:37.932968 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:17:37.933565 13212 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:17:37.933944 13073 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:17:37.934567 13212 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:17:37.942061 10561 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:37.942513 10168 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:37.941332 13215 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:37.947463 8503 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 11 to 12, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:37.950224 10428 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:38.157884 8781 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41858
I20240502 08:17:38.158116 8781 raft_consensus.cc:595] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:38.179728 13241 raft_consensus.cc:1050] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: attempting to promote NON_VOTER 4586652e013946998246ca5489ae7083 to VOTER
I20240502 08:17:38.180495 13241 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:38.184527 10561 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:38.184702 10299 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 LEARNER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:38.184967 8781 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Refusing update from remote peer 78c8d87aa5e44674959cd90af335de6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:17:38.185504 13241 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:38.185904 13188 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:38.186480 13240 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:17:38.199836 13241 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:38.200702 10299 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:38.200919 10561 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:38.204695 8781 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:38.210888 8509 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a reported cstate change: config changed from index 7 to 8, 4586652e013946998246ca5489ae7083 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78c8d87aa5e44674959cd90af335de6a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:38.218756 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:17:38.379606 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49860
I20240502 08:17:38.379844 9041 raft_consensus.cc:595] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:38.401445 8781 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41868
I20240502 08:17:38.401705 8781 raft_consensus.cc:595] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:38.426066 13073 raft_consensus.cc:979] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: : Instructing follower 4f6d482dba7d46b89e70150a61cb5cf0 to start an election
I20240502 08:17:38.426273 13073 raft_consensus.cc:1067] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Signalling peer 4f6d482dba7d46b89e70150a61cb5cf0 to start an election
I20240502 08:17:38.427012 10561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef"
dest_uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
from {username='slave'} at 127.8.68.194:43333
I20240502 08:17:38.427304 10561 raft_consensus.cc:482] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:38.427445 10561 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.429654 10561 raft_consensus.cc:504] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:38.430778 10561 leader_election.cc:288] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [CANDIDATE]: Term 2 election: Requested vote from peers 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207), d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561), eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071)
I20240502 08:17:38.431012 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
I20240502 08:17:38.431228 8781 raft_consensus.cc:3032] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:38.431705 8781 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:17:38.431838 10428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "eae55bf90c524e8eb9df4c31f40096e4"
I20240502 08:17:38.432077 10428 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.431995 8781 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:38.432492 8781 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.436055 8781 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
I20240502 08:17:38.437736 10428 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
I20240502 08:17:38.438277 10497 leader_election.cc:302] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 4f6d482dba7d46b89e70150a61cb5cf0, eae55bf90c524e8eb9df4c31f40096e4; no voters:
I20240502 08:17:38.438637 12357 raft_consensus.cc:2781] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:38.439102 12357 raft_consensus.cc:686] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Becoming Leader. State: Replica: 4f6d482dba7d46b89e70150a61cb5cf0, State: Running, Role: LEADER
I20240502 08:17:38.439527 12357 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:38.442618 8507 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: term changed from 1 to 2, leader changed from 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) to 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200). New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:38.444787 10168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888fe8d0871a4a539b43569e0eda4bef" candidate_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "d1f0504c982c42b8b9de29b8830c1e82"
I20240502 08:17:38.445010 10168 raft_consensus.cc:3037] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.447515 10168 raft_consensus.cc:2454] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6d482dba7d46b89e70150a61cb5cf0 in term 2.
W20240502 08:17:38.517688 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:17:38.594995 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:17:38.621560 9041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620"
dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49880
I20240502 08:17:38.621841 9041 raft_consensus.cc:595] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:38.783385 13287 raft_consensus.cc:979] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: : Instructing follower 14ea87db2f6a4497a2a961afecd23a32 to start an election
I20240502 08:17:38.783591 13188 raft_consensus.cc:1067] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Signalling peer 14ea87db2f6a4497a2a961afecd23a32 to start an election
I20240502 08:17:38.784407 8781 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620"
dest_uuid: "14ea87db2f6a4497a2a961afecd23a32"
from {username='slave'} at 127.8.68.196:35185
I20240502 08:17:38.784699 8781 raft_consensus.cc:482] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:38.784847 8781 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.787267 8781 raft_consensus.cc:504] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:38.788327 8781 leader_election.cc:288] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 election: Requested vote from peers 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907), 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925), 4586652e013946998246ca5489ae7083 (127.8.68.198:36779)
I20240502 08:17:38.788584 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "78c8d87aa5e44674959cd90af335de6a"
I20240502 08:17:38.788771 9041 raft_consensus.cc:3032] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:38.788841 9041 raft_consensus.cc:729] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78c8d87aa5e44674959cd90af335de6a, State: Running, Role: LEADER
I20240502 08:17:38.788796 10561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4f6d482dba7d46b89e70150a61cb5cf0"
I20240502 08:17:38.789036 10561 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.789121 9041 consensus_queue.cc:260] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:38.789578 9041 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.791078 10561 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 2.
I20240502 08:17:38.791456 9041 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 2.
I20240502 08:17:38.791908 8715 leader_election.cc:302] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14ea87db2f6a4497a2a961afecd23a32, 4f6d482dba7d46b89e70150a61cb5cf0, 78c8d87aa5e44674959cd90af335de6a; no voters:
I20240502 08:17:38.792243 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2de1aca9a44a489972aab5decf3620" candidate_uuid: "14ea87db2f6a4497a2a961afecd23a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4586652e013946998246ca5489ae7083"
I20240502 08:17:38.792483 10299 raft_consensus.cc:3037] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.794842 10299 raft_consensus.cc:2454] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea87db2f6a4497a2a961afecd23a32 in term 2.
I20240502 08:17:38.795605 13215 raft_consensus.cc:2781] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:38.795794 13215 raft_consensus.cc:686] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 LEADER]: Becoming Leader. State: Replica: 14ea87db2f6a4497a2a961afecd23a32, State: Running, Role: LEADER
I20240502 08:17:38.796066 13215 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:38.799829 8507 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: term changed from 1 to 2, leader changed from 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) to 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194). New cstate: current_term: 2 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:38.825435 10428 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:38.826445 12357 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 08:17:38.882747 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 08:17:38.884214 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e8896dce80734a7b85155855d6cb3303: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
I20240502 08:17:38.955287 8781 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:38.955940 12357 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:17:38.961431 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:38.962590 13304 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 08:17:39.077765 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:17:39.180339 9041 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:39.180874 13215 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:39.221959 10299 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:39.222560 13215 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:39.228399 10561 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:17:39.231411 13322 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:17:39.552939 10561 consensus_queue.cc:237] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } }
I20240502 08:17:39.555008 10428 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 08:17:39.555554 13347 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 08:17:39.557780 13348 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:39.558362 10428 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:39.559973 10168 raft_consensus.cc:1261] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20240502 08:17:39.560449 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:17:39.561807 13302 consensus_queue.cc:1035] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 08:17:39.562793 8497 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 888fe8d0871a4a539b43569e0eda4bef with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:39.563748 10168 raft_consensus.cc:2932] T 888fe8d0871a4a539b43569e0eda4bef P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } }
I20240502 08:17:39.564046 8508 catalog_manager.cc:5484] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 12 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:39.578869 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet 888fe8d0871a4a539b43569e0eda4bef with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:39.580191 13354 tablet_replica.cc:330] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:39.580396 13354 raft_consensus.cc:2229] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:39.580624 13354 raft_consensus.cc:2258] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:39.581728 13354 ts_tablet_manager.cc:1875] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:39.586385 13354 ts_tablet_manager.cc:1888] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 08:17:39.586504 13354 log.cc:1196] T 888fe8d0871a4a539b43569e0eda4bef P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/888fe8d0871a4a539b43569e0eda4bef
I20240502 08:17:39.587283 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet 888fe8d0871a4a539b43569e0eda4bef (table rebalance_test_table_0 [id=a7c3b6386bf845f0a1ac2b6be0f48325]) successfully deleted
W20240502 08:17:39.714756 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:17:39.751179 10561 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:39.753674 10428 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 08:17:39.754189 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 08:17:39.754309 13348 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 08:17:39.754729 10495 consensus_peers.cc:485] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25df2da40e524e9c95cbe13a484445b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:39.755101 13348 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:17:39.757177 13302 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:39.758390 10428 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:39.762652 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } } }
I20240502 08:17:39.762763 8507 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 11 to 12, NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) added. New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:39.820981 8781 consensus_queue.cc:237] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } }
I20240502 08:17:39.822844 10299 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:39.822885 10561 raft_consensus.cc:1261] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Refusing update from remote peer 14ea87db2f6a4497a2a961afecd23a32: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:17:39.823453 13385 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:39.823776 13384 consensus_queue.cc:1035] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:17:39.826117 13322 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:39.826767 10561 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:39.829555 10299 raft_consensus.cc:2932] T 4e2de1aca9a44a489972aab5decf3620 P 4586652e013946998246ca5489ae7083 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } } }
I20240502 08:17:39.831063 8509 catalog_manager.cc:5484] T 4e2de1aca9a44a489972aab5decf3620 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 8 to 10, VOTER 78c8d87aa5e44674959cd90af335de6a (127.8.68.196) evicted. New cstate: current_term: 2 leader_uuid: "14ea87db2f6a4497a2a961afecd23a32" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4586652e013946998246ca5489ae7083" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 36779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:39.831509 8495 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e2de1aca9a44a489972aab5decf3620 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:39.836077 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2de1aca9a44a489972aab5decf3620 with delete_type TABLET_DATA_TOMBSTONED (TS 78c8d87aa5e44674959cd90af335de6a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52836
I20240502 08:17:39.837018 13387 tablet_replica.cc:330] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: stopping tablet replica
I20240502 08:17:39.837265 13387 raft_consensus.cc:2229] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:39.837462 13387 raft_consensus.cc:2258] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:39.838779 13387 ts_tablet_manager.cc:1875] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:39.845654 13387 ts_tablet_manager.cc:1888] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:17:39.845836 13387 log.cc:1196] T 4e2de1aca9a44a489972aab5decf3620 P 78c8d87aa5e44674959cd90af335de6a: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/4e2de1aca9a44a489972aab5decf3620
I20240502 08:17:39.847080 8494 catalog_manager.cc:4830] TS 78c8d87aa5e44674959cd90af335de6a (127.8.68.196:35907): tablet 4e2de1aca9a44a489972aab5decf3620 (table rebalance_test_table_1 [id=518fb706e1334f4eb179a5e0f5f22392]) successfully deleted
I20240502 08:17:40.163620 13403 ts_tablet_manager.cc:902] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Initiating tablet copy from peer 4f6d482dba7d46b89e70150a61cb5cf0 (127.8.68.200:32925)
I20240502 08:17:40.165566 13403 tablet_copy_client.cc:308] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.200:32925
I20240502 08:17:40.171857 10582 tablet_copy_service.cc:140] P 4f6d482dba7d46b89e70150a61cb5cf0: Received BeginTabletCopySession request for tablet 25df2da40e524e9c95cbe13a484445b5 from peer d1f0504c982c42b8b9de29b8830c1e82 ({username='slave'} at 127.8.68.197:39233)
I20240502 08:17:40.172101 10582 tablet_copy_service.cc:161] P 4f6d482dba7d46b89e70150a61cb5cf0: Beginning new tablet copy session on tablet 25df2da40e524e9c95cbe13a484445b5 from peer d1f0504c982c42b8b9de29b8830c1e82 at {username='slave'} at 127.8.68.197:39233: session id = d1f0504c982c42b8b9de29b8830c1e82-25df2da40e524e9c95cbe13a484445b5
I20240502 08:17:40.175312 10582 tablet_copy_source_session.cc:205] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:40.177198 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25df2da40e524e9c95cbe13a484445b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.182404 13403 tablet_copy_client.cc:783] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:40.182662 13403 tablet_copy_client.cc:647] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:40.185070 13407 tablet_copy_client.cc:1002] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Injecting latency into file download: 1500
W20240502 08:17:40.766412 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:40.908684 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:41.166944 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:41.463474 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:17:41.685530 13403 tablet_copy_client.cc:523] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:41.688266 13403 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap starting.
I20240502 08:17:41.730104 13403 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:41.730515 13403 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Bootstrap complete.
I20240502 08:17:41.730748 13403 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Time spent bootstrapping tablet: real 0.043s user 0.035s sys 0.004s
I20240502 08:17:41.731390 13403 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:41.731654 13403 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1f0504c982c42b8b9de29b8830c1e82, State: Initialized, Role: LEARNER
I20240502 08:17:41.731848 13403 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: true } }
I20240502 08:17:41.732637 13403 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:41.733522 10582 tablet_copy_service.cc:342] P 4f6d482dba7d46b89e70150a61cb5cf0: Request end of tablet copy session d1f0504c982c42b8b9de29b8830c1e82-25df2da40e524e9c95cbe13a484445b5 received from {username='slave'} at 127.8.68.197:39233
I20240502 08:17:41.733690 10582 tablet_copy_service.cc:433] P 4f6d482dba7d46b89e70150a61cb5cf0: ending tablet copy session d1f0504c982c42b8b9de29b8830c1e82-25df2da40e524e9c95cbe13a484445b5 on tablet 25df2da40e524e9c95cbe13a484445b5 with peer d1f0504c982c42b8b9de29b8830c1e82
I20240502 08:17:41.897734 10168 raft_consensus.cc:1203] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240502 08:17:42.031109 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25df2da40e524e9c95cbe13a484445b5: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:42.167462 8717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:42.168437 8717 consensus_peers.cc:485] T e606e24b29c04377ad5eb8576dc4c370 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:17:42.220288 8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:17:42.292618 8717 consensus_peers.cc:485] T 5553c68dc6b24e5bbd849453a0692a73 P 14ea87db2f6a4497a2a961afecd23a32 -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:17:42.345527 8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:17:42.523401 13469 raft_consensus.cc:1050] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0: attempting to promote NON_VOTER d1f0504c982c42b8b9de29b8830c1e82 to VOTER
I20240502 08:17:42.524292 13469 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:42.526556 10427 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:42.526829 10168 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 LEARNER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:42.526947 8781 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:17:42.527385 13469 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:17:42.527850 13472 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:17:42.528267 13470 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:17:42.532071 13469 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.532825 10168 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.532871 8781 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.537475 8507 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 reported cstate change: config changed from index 12 to 13, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.540966 10427 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.556242 10562 consensus_queue.cc:237] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } }
I20240502 08:17:42.558714 10427 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 08:17:42.559612 13469 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 08:17:42.559669 10168 raft_consensus.cc:1261] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Refusing update from remote peer 4f6d482dba7d46b89e70150a61cb5cf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 08:17:42.560621 13469 consensus_queue.cc:1035] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 08:17:42.562458 13470 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.563094 10427 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P eae55bf90c524e8eb9df4c31f40096e4 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.563589 10167 raft_consensus.cc:2932] T 25df2da40e524e9c95cbe13a484445b5 P d1f0504c982c42b8b9de29b8830c1e82 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } } }
I20240502 08:17:42.567574 8497 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25df2da40e524e9c95cbe13a484445b5 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:42.568663 8509 catalog_manager.cc:5484] T 25df2da40e524e9c95cbe13a484445b5 P 4f6d482dba7d46b89e70150a61cb5cf0 reported cstate change: config changed from index 13 to 14, VOTER 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae55bf90c524e8eb9df4c31f40096e4" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 45071 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f0504c982c42b8b9de29b8830c1e82" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:42.573230 8760 tablet_service.cc:1514] Processing DeleteTablet for tablet 25df2da40e524e9c95cbe13a484445b5 with delete_type TABLET_DATA_TOMBSTONED (TS 14ea87db2f6a4497a2a961afecd23a32 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35894
I20240502 08:17:42.574020 13487 tablet_replica.cc:330] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: stopping tablet replica
I20240502 08:17:42.574287 13487 raft_consensus.cc:2229] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:42.574698 13487 raft_consensus.cc:2258] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:42.577857 13487 ts_tablet_manager.cc:1875] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:42.584009 13487 ts_tablet_manager.cc:1888] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 08:17:42.584126 13487 log.cc:1196] T 25df2da40e524e9c95cbe13a484445b5 P 14ea87db2f6a4497a2a961afecd23a32: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/25df2da40e524e9c95cbe13a484445b5
I20240502 08:17:42.584934 8495 catalog_manager.cc:4830] TS 14ea87db2f6a4497a2a961afecd23a32 (127.8.68.194:46207): tablet 25df2da40e524e9c95cbe13a484445b5 (table rebalance_test_table_2 [id=283e6f789950422eb0de42bdc59af5e5]) successfully deleted
I20240502 08:17:42.685881 8760 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:42.703863 10542 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:42.715387 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:42.725757 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:42.748328 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:42.786902 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:42.801246 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:42.955965 8760 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:42.998797 10542 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:42.999303 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:43.006284 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:43.015193 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:43.019151 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:43.029196 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:43.306537 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:34415
--local_ip_for_outbound_sockets=127.8.68.193
--tserver_master_addrs=127.8.68.254:40717
--webserver_port=46605
--webserver_interface=127.8.68.193
--builtin_ntp_servers=127.8.68.212:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 08:17:43.389261 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:17:43.431497 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:17:43.658264 13559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:43.658639 13559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:43.658809 13559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:43.667398 13559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:43.667541 13559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:43.667706 13559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240502 08:17:43.677567 13559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:34415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=46605
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.8.68.254:40717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:43.678974 13559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:43.680806 13559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:43.694933 13567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:43.695533 13566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:43.696345 13569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:43.697494 13559 server_base.cc:1034] running on GCE node
I20240502 08:17:43.698171 13559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:43.699301 13559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:43.700541 13559 hybrid_clock.cc:648] HybridClock initialized: now 1714637863700477 us; error 56 us; skew 500 ppm
I20240502 08:17:43.700908 13559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:43.703446 13559 webserver.cc:460] Webserver started at http://127.8.68.193:46605/ using document root <none> and password file <none>
I20240502 08:17:43.704243 13559 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:43.704381 13559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:43.710047 13559 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 08:17:43.714355 13575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:43.715987 13559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 08:17:43.716159 13559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bdf1a2f21da04697b58f19f331492b84"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-2dhs"
I20240502 08:17:43.717155 13559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:43.740253 13559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:43.741075 13559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:43.741357 13559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:43.742302 13559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:43.744650 13582 ts_tablet_manager.cc:522] Loading tablet metadata (0/11 complete)
I20240502 08:17:43.768306 13559 ts_tablet_manager.cc:559] Loaded tablet metadata (11 total tablets, 11 live tablets)
I20240502 08:17:43.768429 13559 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.024s user 0.002s sys 0.000s
I20240502 08:17:43.768544 13559 ts_tablet_manager.cc:574] Registering tablets (0/11 complete)
I20240502 08:17:43.771986 13582 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:43.788046 13559 ts_tablet_manager.cc:590] Registered 11 tablets
I20240502 08:17:43.788178 13559 ts_tablet_manager.cc:569] Time spent register tablets: real 0.020s user 0.016s sys 0.003s
I20240502 08:17:43.791119 13582 log.cc:824] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:43.823161 13559 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:34415
I20240502 08:17:43.823292 13689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:34415 every 8 connection(s)
I20240502 08:17:43.824496 13559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:43.826418 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 13559
I20240502 08:17:43.851943 13690 heartbeater.cc:347] Connected to a master server at 127.8.68.254:40717
I20240502 08:17:43.852296 13690 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:43.853482 13690 heartbeater.cc:511] Master 127.8.68.254:40717 requested a full tablet report, sending...
I20240502 08:17:43.857160 8507 ts_manager.cc:194] Re-registered known tserver with Master: bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415)
I20240502 08:17:43.861538 8507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:58607
I20240502 08:17:43.870224 13582 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:43.870801 13582 tablet_bootstrap.cc:492] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:43.872557 13582 ts_tablet_manager.cc:1367] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.101s user 0.063s sys 0.012s
W20240502 08:17:43.874250 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240502 08:17:43.877489 13582 raft_consensus.cc:348] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:43.878214 13582 raft_consensus.cc:729] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:43.878846 13582 consensus_queue.cc:260] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:43.879969 13690 heartbeater.cc:503] Master 127.8.68.254:40717 was elected leader, sending a full tablet report...
I20240502 08:17:43.880370 13582 ts_tablet_manager.cc:1398] T e8896dce80734a7b85155855d6cb3303 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 08:17:43.880748 13582 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:43.933254 13582 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:43.933862 13582 tablet_bootstrap.cc:492] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:43.935168 13582 ts_tablet_manager.cc:1367] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.054s user 0.050s sys 0.000s
I20240502 08:17:43.936357 13582 raft_consensus.cc:348] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:43.936616 13582 raft_consensus.cc:729] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:43.936898 13582 consensus_queue.cc:260] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:43.937587 13582 ts_tablet_manager.cc:1398] T 171c663b69ed42f9883aca8e7392cded P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:43.937871 13582 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
W20240502 08:17:43.985898 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240502 08:17:43.993928 13582 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:43.994513 13582 tablet_bootstrap.cc:492] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:43.995903 13582 ts_tablet_manager.cc:1367] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.008s
I20240502 08:17:43.996704 13582 raft_consensus.cc:348] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:17:43.996935 13582 raft_consensus.cc:729] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:43.997220 13582 consensus_queue.cc:260] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } }
I20240502 08:17:43.997958 13582 ts_tablet_manager.cc:1398] T 5553c68dc6b24e5bbd849453a0692a73 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:43.998255 13582 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.042546 13582 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.043069 13582 tablet_bootstrap.cc:492] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.044123 13582 ts_tablet_manager.cc:1367] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.046s user 0.033s sys 0.012s
I20240502 08:17:44.044858 13582 raft_consensus.cc:348] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.045028 13582 raft_consensus.cc:729] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.045257 13582 consensus_queue.cc:260] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.045714 13582 ts_tablet_manager.cc:1398] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:44.046022 13582 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.083024 13582 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.083621 13582 tablet_bootstrap.cc:492] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.084690 13582 ts_tablet_manager.cc:1367] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.039s user 0.037s sys 0.000s
I20240502 08:17:44.085341 13582 raft_consensus.cc:348] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.085525 13582 raft_consensus.cc:729] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.085800 13582 consensus_queue.cc:260] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.086246 13582 ts_tablet_manager.cc:1398] T 295aa577b3bf4bfebf8870b8a78aaa12 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:44.086505 13582 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.126076 13582 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.126616 13582 tablet_bootstrap.cc:492] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.127676 13582 ts_tablet_manager.cc:1367] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.023s
I20240502 08:17:44.128319 13582 raft_consensus.cc:348] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.128486 13582 raft_consensus.cc:729] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.128720 13582 consensus_queue.cc:260] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.129182 13582 ts_tablet_manager.cc:1398] T 3b8f8d5081284a3288dab20e0e19d9bf P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:44.129505 13582 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.169296 13582 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.169780 13582 tablet_bootstrap.cc:492] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.170794 13582 ts_tablet_manager.cc:1367] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.016s
I20240502 08:17:44.171465 13582 raft_consensus.cc:348] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:17:44.171658 13582 raft_consensus.cc:729] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.171933 13582 consensus_queue.cc:260] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } }
I20240502 08:17:44.172428 13582 ts_tablet_manager.cc:1398] T 25df2da40e524e9c95cbe13a484445b5 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:17:44.172777 13582 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.216097 13645 raft_consensus.cc:3037] T e606e24b29c04377ad5eb8576dc4c370 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:44.237531 13582 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.238144 13582 tablet_bootstrap.cc:492] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.239538 13582 ts_tablet_manager.cc:1367] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.067s user 0.050s sys 0.016s
I20240502 08:17:44.240433 13582 raft_consensus.cc:348] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:44.240667 13582 raft_consensus.cc:729] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.240939 13582 consensus_queue.cc:260] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:44.241456 13582 ts_tablet_manager.cc:1398] T 49af926b61e64eb0ac9c4dcd5b638152 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:44.241747 13582 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.300886 13582 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.301448 13582 tablet_bootstrap.cc:492] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.302743 13582 ts_tablet_manager.cc:1367] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.061s user 0.046s sys 0.012s
I20240502 08:17:44.303453 13582 raft_consensus.cc:348] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:44.303684 13582 raft_consensus.cc:729] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.303915 13582 consensus_queue.cc:260] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } attrs { replace: true } } peers { permanent_uuid: "4f6d482dba7d46b89e70150a61cb5cf0" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 32925 } attrs { promote: true } }
I20240502 08:17:44.304567 13582 ts_tablet_manager.cc:1398] T 888fe8d0871a4a539b43569e0eda4bef P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:44.304839 13582 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.359695 13582 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.360306 13582 tablet_bootstrap.cc:492] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.361655 13582 ts_tablet_manager.cc:1367] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.057s user 0.042s sys 0.012s
I20240502 08:17:44.362594 13582 raft_consensus.cc:348] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:44.362820 13582 raft_consensus.cc:729] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.363075 13582 consensus_queue.cc:260] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c8d87aa5e44674959cd90af335de6a" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 35907 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } }
I20240502 08:17:44.363605 13582 ts_tablet_manager.cc:1398] T 41873909d8db44f89580fb209502ab30 P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 08:17:44.363961 13582 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Bootstrap starting.
I20240502 08:17:44.418398 13582 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:44.418959 13582 tablet_bootstrap.cc:492] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Bootstrap complete.
I20240502 08:17:44.420107 13582 ts_tablet_manager.cc:1367] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Time spent bootstrapping tablet: real 0.056s user 0.044s sys 0.009s
I20240502 08:17:44.420799 13582 raft_consensus.cc:348] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.421103 13582 raft_consensus.cc:729] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf1a2f21da04697b58f19f331492b84, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.421417 13582 consensus_queue.cc:260] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ea87db2f6a4497a2a961afecd23a32" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46207 } } peers { permanent_uuid: "bdf1a2f21da04697b58f19f331492b84" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 34415 } } peers { permanent_uuid: "71df2128b229478facd8ba1b2683ccca" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 36747 } }
I20240502 08:17:44.422065 13582 ts_tablet_manager.cc:1398] T c9479bf73bb1400c83c247f3654bb79b P bdf1a2f21da04697b58f19f331492b84: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:44.486305 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.496487 8760 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:44.521888 13625 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.525540 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.544754 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:44.559036 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:44.561530 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.589751 10542 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:44.776511 10542 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:44.790285 10148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.792667 8760 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:44.796939 13625 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.800719 10279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.800796 8893 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:44.831918 9021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:44.850194 10409 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:45.050485 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 13559
I20240502 08:17:45.057909 13685 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:45.167361 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 13559
I20240502 08:17:45.190090 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 8700
I20240502 08:17:45.196738 8820 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:45.202836 8496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 08:17:45.204111 8496 catalog_manager.cc:4573] TS bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 49af926b61e64eb0ac9c4dcd5b638152: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111)
W20240502 08:17:45.299005 8848 consensus_peers.cc:485] T 41873909d8db44f89580fb209502ab30 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:45.303035 8978 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:45.407590 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 8700
W20240502 08:17:45.443866 10496 connection.cc:535] server connection from 127.8.68.194:43333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:45.444274 10235 connection.cc:535] server connection from 127.8.68.194:50299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:45.444983 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 8831
W20240502 08:17:45.445035 8976 connection.cc:535] server connection from 127.8.68.194:57701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:45.451457 8953 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:45.525760 8978 consensus_peers.cc:485] T 3b8f8d5081284a3288dab20e0e19d9bf P 78c8d87aa5e44674959cd90af335de6a -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:45.633882 8848 consensus_peers.cc:485] T 295aa577b3bf4bfebf8870b8a78aaa12 P 71df2128b229478facd8ba1b2683ccca -> Peer bdf1a2f21da04697b58f19f331492b84 (127.8.68.193:34415): Couldn't send request to peer bdf1a2f21da04697b58f19f331492b84. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:34415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:45.634284 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 8831
W20240502 08:17:45.667917 8976 connection.cc:535] server connection from 127.8.68.195:53447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:45.668176 8976 connection.cc:535] client connection to 127.8.68.195:36747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:45.668596 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 8961
W20240502 08:17:45.669179 8976 consensus_peers.cc:485] T 171c663b69ed42f9883aca8e7392cded P 78c8d87aa5e44674959cd90af335de6a -> Peer 71df2128b229478facd8ba1b2683ccca (127.8.68.195:36747): Couldn't send request to peer 71df2128b229478facd8ba1b2683ccca. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.195:36747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:45.674355 9082 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:45.861333 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 8961
W20240502 08:17:45.896193 8497 connection.cc:535] server connection from 127.8.68.196:50311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:45.896605 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 10086
I20240502 08:17:45.903491 10206 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:46.035449 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 10086
I20240502 08:17:46.065447 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 10217
I20240502 08:17:46.072371 10337 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:46.087555 10495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:37561: connect: Connection refused (error 111)
W20240502 08:17:46.089143 10495 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 -> Peer d1f0504c982c42b8b9de29b8830c1e82 (127.8.68.197:37561): Couldn't send request to peer d1f0504c982c42b8b9de29b8830c1e82. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:37561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:46.205507 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 10217
I20240502 08:17:46.229560 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 10347
I20240502 08:17:46.236027 10469 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:46.368034 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 10347
W20240502 08:17:46.393097 10497 connection.cc:535] client connection to 127.8.68.199:45071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:46.393484 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 10478
W20240502 08:17:46.395052 10497 consensus_peers.cc:485] T 888fe8d0871a4a539b43569e0eda4bef P 4f6d482dba7d46b89e70150a61cb5cf0 -> Peer eae55bf90c524e8eb9df4c31f40096e4 (127.8.68.199:45071): Couldn't send request to peer eae55bf90c524e8eb9df4c31f40096e4. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:45071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:46.401180 10601 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:46.542610 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 10478
I20240502 08:17:46.568869 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 8479
I20240502 08:17:46.574951 8539 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:46.744809 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 8479
2024-05-02T08:17:46Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (71167 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (71167 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 08:17:46.821983 8467 test_util.cc:268] Using random seed: 1022912826
I20240502 08:17:46.823952 8467 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:46.824048 8467 ts_itest-base.cc:113] --------------
I20240502 08:17:46.824106 8467 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:17:46.824174 8467 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:17:46.824234 8467 ts_itest-base.cc:116] --------------
2024-05-02T08:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:46Z Disabled control of system clock
I20240502 08:17:46.849259 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:43983
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:43983
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 08:17:47.178778 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:47.179096 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:47.179222 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:47.186786 13836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:47.186918 13836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:47.186964 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:47.187057 13836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:47.187129 13836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:47.196341 13836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:43983
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:47.197703 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:47.199560 13836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:47.210772 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:47.211627 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:47.212463 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:47.213383 13836 server_base.cc:1034] running on GCE node
I20240502 08:17:47.214175 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:47.215721 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:47.216933 13836 hybrid_clock.cc:648] HybridClock initialized: now 1714637867216877 us; error 47 us; skew 500 ppm
I20240502 08:17:47.217293 13836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:47.224244 13836 webserver.cc:460] Webserver started at http://127.8.68.254:37907/ using document root <none> and password file <none>
I20240502 08:17:47.225060 13836 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:47.225188 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:47.225525 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:47.228365 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8a82cc1b6ec54cfda308f92ac8deba7e"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.229161 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8a82cc1b6ec54cfda308f92ac8deba7e"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.234740 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240502 08:17:47.238552 13850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:47.240159 13836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:47.240360 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8a82cc1b6ec54cfda308f92ac8deba7e"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.240574 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:47.262262 13836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:47.263378 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:47.263655 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:47.282989 13836 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:43983
I20240502 08:17:47.283008 13901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:43983 every 8 connection(s)
I20240502 08:17:47.284433 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:47.288038 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 13836
I20240502 08:17:47.288348 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:47.288847 13902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.300704 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e: Bootstrap starting.
I20240502 08:17:47.304417 13902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.305415 13902 log.cc:824] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:47.308007 13902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e: No bootstrap required, opened a new log
I20240502 08:17:47.314455 13902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } }
I20240502 08:17:47.314831 13902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.314930 13902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a82cc1b6ec54cfda308f92ac8deba7e, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.315558 13902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } }
I20240502 08:17:47.315765 13902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:47.315888 13902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:47.316032 13902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.318235 13902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } }
I20240502 08:17:47.318727 13902 leader_election.cc:302] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a82cc1b6ec54cfda308f92ac8deba7e; no voters:
I20240502 08:17:47.319355 13902 leader_election.cc:288] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:47.319533 13907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:47.320173 13907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [term 1 LEADER]: Becoming Leader. State: Replica: 8a82cc1b6ec54cfda308f92ac8deba7e, State: Running, Role: LEADER
I20240502 08:17:47.320747 13907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } }
I20240502 08:17:47.321314 13902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:47.323932 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } } }
I20240502 08:17:47.324414 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:47.324051 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a82cc1b6ec54cfda308f92ac8deba7e. Latest consensus state: current_term: 1 leader_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a82cc1b6ec54cfda308f92ac8deba7e" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43983 } } }
I20240502 08:17:47.325559 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:47.327957 13915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:47.332536 13915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:47.341116 13915 catalog_manager.cc:1347] Generated new cluster ID: d8932476f17948d3ba8ff3ef715ddfff
I20240502 08:17:47.341262 13915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:47.358991 13915 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:47.360316 13915 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:47.376194 13915 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8a82cc1b6ec54cfda308f92ac8deba7e: Generated new TSK 0
I20240502 08:17:47.377197 13915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:47.382579 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:17:47.718677 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:47.718973 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:47.719097 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:47.726891 13926 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:47.727023 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:47.727190 13926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240502 08:17:47.736274 13926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:47.737612 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:47.739404 13926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:47.751772 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:47.752718 13932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:47.753508 13934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:47.754477 13926 server_base.cc:1034] running on GCE node
I20240502 08:17:47.755101 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:47.756114 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:47.757337 13926 hybrid_clock.cc:648] HybridClock initialized: now 1714637867757302 us; error 58 us; skew 500 ppm
I20240502 08:17:47.757668 13926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:47.759974 13926 webserver.cc:460] Webserver started at http://127.8.68.193:36441/ using document root <none> and password file <none>
I20240502 08:17:47.760697 13926 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:47.760803 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:47.761116 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:47.763731 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6645b5beb31941ffa7658eff74f7be28"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.764496 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6645b5beb31941ffa7658eff74f7be28"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.769709 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:47.773253 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:47.774626 13926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:47.774796 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6645b5beb31941ffa7658eff74f7be28"
format_stamp: "Formatted at 2024-05-02 08:17:47 on dist-test-slave-2dhs"
I20240502 08:17:47.774999 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:47.795559 13926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:47.796458 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:47.796736 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:47.797793 13926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:47.799383 13926 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:47.799481 13926 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:47.799600 13926 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:47.799674 13926 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:47.834187 13926 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:41369
I20240502 08:17:47.834245 14052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:41369 every 8 connection(s)
I20240502 08:17:47.835578 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:47.837982 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 13926
I20240502 08:17:47.838207 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:47.841923 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:47.850040 14053 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43983
I20240502 08:17:47.850360 14053 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:47.851078 14053 heartbeater.cc:511] Master 127.8.68.254:43983 requested a full tablet report, sending...
I20240502 08:17:47.853349 13867 ts_manager.cc:194] Registered new tserver with Master: 6645b5beb31941ffa7658eff74f7be28 (127.8.68.193:41369)
I20240502 08:17:47.855681 13867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:41717
W20240502 08:17:48.179302 14057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:48.179608 14057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:48.179751 14057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:48.187170 14057 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:48.187369 14057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:48.187562 14057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240502 08:17:48.196605 14057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:48.197961 14057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:48.199764 14057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:48.212489 14063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:48.213089 14064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:48.213822 14067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:48.214846 14057 server_base.cc:1034] running on GCE node
I20240502 08:17:48.215664 14057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:48.216709 14057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:48.217986 14057 hybrid_clock.cc:648] HybridClock initialized: now 1714637868217935 us; error 51 us; skew 500 ppm
I20240502 08:17:48.218394 14057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:48.220772 14057 webserver.cc:460] Webserver started at http://127.8.68.194:37757/ using document root <none> and password file <none>
I20240502 08:17:48.221642 14057 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:48.221799 14057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:48.222118 14057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:48.224786 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "322ebe17500347659bae3076c33d9c5d"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.225556 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "322ebe17500347659bae3076c33d9c5d"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.230787 14057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240502 08:17:48.234306 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.235699 14057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:48.235860 14057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "322ebe17500347659bae3076c33d9c5d"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.236069 14057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:48.282039 14057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:48.283130 14057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:48.283430 14057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:48.284420 14057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:48.285902 14057 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:48.285987 14057 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.286095 14057 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:48.286162 14057 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.316923 14057 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:44407
I20240502 08:17:48.316992 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:44407 every 8 connection(s)
I20240502 08:17:48.318260 14057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:48.323401 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14057
I20240502 08:17:48.323606 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:48.326957 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:48.331382 14187 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43983
I20240502 08:17:48.331657 14187 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:48.332384 14187 heartbeater.cc:511] Master 127.8.68.254:43983 requested a full tablet report, sending...
I20240502 08:17:48.333808 13867 ts_manager.cc:194] Registered new tserver with Master: 322ebe17500347659bae3076c33d9c5d (127.8.68.194:44407)
I20240502 08:17:48.334573 13867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:33505
W20240502 08:17:48.660912 14190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:48.661250 14190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:48.661403 14190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:48.669113 14190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:48.669279 14190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:48.669445 14190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240502 08:17:48.678647 14190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:48.680116 14190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:48.681905 14190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:48.694612 14196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:48.695024 14195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:48.695879 14198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:48.696739 14190 server_base.cc:1034] running on GCE node
I20240502 08:17:48.697435 14190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:48.698470 14190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:48.699671 14190 hybrid_clock.cc:648] HybridClock initialized: now 1714637868699615 us; error 44 us; skew 500 ppm
I20240502 08:17:48.700006 14190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:48.702435 14190 webserver.cc:460] Webserver started at http://127.8.68.195:35269/ using document root <none> and password file <none>
I20240502 08:17:48.703156 14190 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:48.703286 14190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:48.703593 14190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:48.706295 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a455c7c4dcc14281bf26960d09d2edad"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.707089 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a455c7c4dcc14281bf26960d09d2edad"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.712520 14190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 08:17:48.716181 14204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.717590 14190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:48.717738 14190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a455c7c4dcc14281bf26960d09d2edad"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-2dhs"
I20240502 08:17:48.717927 14190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:48.742475 14190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:48.743396 14190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:48.743644 14190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:48.744604 14190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:48.746248 14190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:48.746351 14190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.746440 14190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:48.746495 14190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:48.779027 14190 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:40509
I20240502 08:17:48.779083 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:40509 every 8 connection(s)
I20240502 08:17:48.780592 14190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:48.787947 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14190
I20240502 08:17:48.788113 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:48.791410 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:48.793562 14317 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43983
I20240502 08:17:48.793879 14317 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:48.794642 14317 heartbeater.cc:511] Master 127.8.68.254:43983 requested a full tablet report, sending...
I20240502 08:17:48.795979 13867 ts_manager.cc:194] Registered new tserver with Master: a455c7c4dcc14281bf26960d09d2edad (127.8.68.195:40509)
I20240502 08:17:48.796774 13867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:46723
I20240502 08:17:48.858952 14053 heartbeater.cc:503] Master 127.8.68.254:43983 was elected leader, sending a full tablet report...
W20240502 08:17:49.138392 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:49.138737 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:49.138886 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:49.146709 14320 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:49.146870 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:49.147042 14320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240502 08:17:49.156711 14320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:49.158054 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:49.159817 14320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:49.172335 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:49.172783 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:49.173792 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:49.174548 14320 server_base.cc:1034] running on GCE node
I20240502 08:17:49.175168 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:49.176172 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:49.177390 14320 hybrid_clock.cc:648] HybridClock initialized: now 1714637869177340 us; error 46 us; skew 500 ppm
I20240502 08:17:49.177714 14320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:49.180131 14320 webserver.cc:460] Webserver started at http://127.8.68.196:39945/ using document root <none> and password file <none>
I20240502 08:17:49.180872 14320 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:49.180992 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:49.181298 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:49.183974 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1655c794693a4259a9bf2901ae7204b1"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.184728 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1655c794693a4259a9bf2901ae7204b1"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.189692 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:49.192972 14334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.194252 14320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:49.194412 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1655c794693a4259a9bf2901ae7204b1"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.194604 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:49.207993 14320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:49.208811 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:49.209069 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:49.209964 14320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:49.211459 14320 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:49.211556 14320 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.211668 14320 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:49.211740 14320 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.244108 14320 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:41659
I20240502 08:17:49.244167 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:41659 every 8 connection(s)
I20240502 08:17:49.245469 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:49.253537 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14320
I20240502 08:17:49.253728 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:49.257143 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--builtin_ntp_servers=127.8.68.212:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:49.258481 14447 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43983
I20240502 08:17:49.258827 14447 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:49.259714 14447 heartbeater.cc:511] Master 127.8.68.254:43983 requested a full tablet report, sending...
I20240502 08:17:49.261094 13867 ts_manager.cc:194] Registered new tserver with Master: 1655c794693a4259a9bf2901ae7204b1 (127.8.68.196:41659)
I20240502 08:17:49.261929 13867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:48705
I20240502 08:17:49.337476 14187 heartbeater.cc:503] Master 127.8.68.254:43983 was elected leader, sending a full tablet report...
W20240502 08:17:49.586863 14450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:49.587153 14450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:49.587307 14450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:49.594660 14450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:49.594789 14450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:49.594959 14450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:17:49.604104 14450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:49.605458 14450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:49.607194 14450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:49.619988 14456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:49.620543 14455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:49.621394 14458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:49.622284 14450 server_base.cc:1034] running on GCE node
I20240502 08:17:49.623064 14450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:49.624130 14450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:49.625342 14450 hybrid_clock.cc:648] HybridClock initialized: now 1714637869625306 us; error 64 us; skew 500 ppm
I20240502 08:17:49.625671 14450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:49.628079 14450 webserver.cc:460] Webserver started at http://127.8.68.197:36407/ using document root <none> and password file <none>
I20240502 08:17:49.628858 14450 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:49.628980 14450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:49.629307 14450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:49.632025 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3c1c47633c3a4895a150cbd01b5154f0"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.632812 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3c1c47633c3a4895a150cbd01b5154f0"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.638026 14450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:49.641592 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.642961 14450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:49.643126 14450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3c1c47633c3a4895a150cbd01b5154f0"
format_stamp: "Formatted at 2024-05-02 08:17:49 on dist-test-slave-2dhs"
I20240502 08:17:49.643376 14450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:49.665638 14450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:49.666513 14450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:49.666797 14450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:49.667783 14450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:49.669301 14450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:49.669502 14450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.669667 14450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:49.669773 14450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:49.702922 14450 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:35155
I20240502 08:17:49.702971 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:35155 every 8 connection(s)
I20240502 08:17:49.704320 14450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:49.706420 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14450
I20240502 08:17:49.706627 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:49.717340 14577 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43983
I20240502 08:17:49.717629 14577 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:49.718353 14577 heartbeater.cc:511] Master 127.8.68.254:43983 requested a full tablet report, sending...
I20240502 08:17:49.719682 13867 ts_manager.cc:194] Registered new tserver with Master: 3c1c47633c3a4895a150cbd01b5154f0 (127.8.68.197:35155)
I20240502 08:17:49.720434 13867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:57189
I20240502 08:17:49.722740 8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:17:49.738746 13867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42808:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:17:49.783445 14252 tablet_service.cc:1467] Processing CreateTablet for tablet 72f31bf9d6d54b589c5b513238d6b33c (DEFAULT_TABLE table=TestTable [id=8ace7cb308524f5299c0e8024b237293]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.782078 14122 tablet_service.cc:1467] Processing CreateTablet for tablet 72f31bf9d6d54b589c5b513238d6b33c (DEFAULT_TABLE table=TestTable [id=8ace7cb308524f5299c0e8024b237293]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.785115 14122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f31bf9d6d54b589c5b513238d6b33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.785141 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f31bf9d6d54b589c5b513238d6b33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.788498 13988 tablet_service.cc:1467] Processing CreateTablet for tablet 72f31bf9d6d54b589c5b513238d6b33c (DEFAULT_TABLE table=TestTable [id=8ace7cb308524f5299c0e8024b237293]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.791405 13988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f31bf9d6d54b589c5b513238d6b33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.799352 14596 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Bootstrap starting.
I20240502 08:17:49.799746 14317 heartbeater.cc:503] Master 127.8.68.254:43983 was elected leader, sending a full tablet report...
I20240502 08:17:49.804018 14596 tablet_bootstrap.cc:654] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.804910 14595 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Bootstrap starting.
I20240502 08:17:49.806694 14597 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Bootstrap starting.
I20240502 08:17:49.807812 14596 log.cc:824] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:49.808370 14595 tablet_bootstrap.cc:654] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.809513 14595 log.cc:824] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:49.810096 14597 tablet_bootstrap.cc:654] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.811336 14596 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: No bootstrap required, opened a new log
I20240502 08:17:49.811318 14597 log.cc:824] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:49.811627 14596 ts_tablet_manager.cc:1367] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240502 08:17:49.812018 14595 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: No bootstrap required, opened a new log
I20240502 08:17:49.812274 14595 ts_tablet_manager.cc:1367] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:17:49.818929 14595 raft_consensus.cc:348] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.819327 14595 raft_consensus.cc:374] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.819466 14595 raft_consensus.cc:729] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322ebe17500347659bae3076c33d9c5d, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.819998 14597 tablet_bootstrap.cc:492] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: No bootstrap required, opened a new log
I20240502 08:17:49.820086 14595 consensus_queue.cc:260] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.820286 14597 ts_tablet_manager.cc:1367] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 08:17:49.822013 14595 ts_tablet_manager.cc:1398] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20240502 08:17:49.821826 14596 raft_consensus.cc:348] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.822420 14596 raft_consensus.cc:374] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.822532 14596 raft_consensus.cc:729] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a455c7c4dcc14281bf26960d09d2edad, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.823340 14596 consensus_queue.cc:260] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.825065 14596 ts_tablet_manager.cc:1398] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240502 08:17:49.828783 14597 raft_consensus.cc:348] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.829149 14597 raft_consensus.cc:374] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.829241 14597 raft_consensus.cc:729] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6645b5beb31941ffa7658eff74f7be28, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.829872 14597 consensus_queue.cc:260] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:49.831467 14597 ts_tablet_manager.cc:1398] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20240502 08:17:49.838490 14054 tablet.cc:2369] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:50.033011 14318 tablet.cc:2369] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:50.049011 14602 raft_consensus.cc:482] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:50.049314 14602 raft_consensus.cc:504] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:50.051051 14602 leader_election.cc:288] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 322ebe17500347659bae3076c33d9c5d (127.8.68.194:44407), 6645b5beb31941ffa7658eff74f7be28 (127.8.68.193:41369)
I20240502 08:17:50.058491 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f31bf9d6d54b589c5b513238d6b33c" candidate_uuid: "a455c7c4dcc14281bf26960d09d2edad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322ebe17500347659bae3076c33d9c5d" is_pre_election: true
I20240502 08:17:50.058825 14008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f31bf9d6d54b589c5b513238d6b33c" candidate_uuid: "a455c7c4dcc14281bf26960d09d2edad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6645b5beb31941ffa7658eff74f7be28" is_pre_election: true
I20240502 08:17:50.059110 14142 raft_consensus.cc:2454] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a455c7c4dcc14281bf26960d09d2edad in term 0.
I20240502 08:17:50.059345 14008 raft_consensus.cc:2454] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a455c7c4dcc14281bf26960d09d2edad in term 0.
I20240502 08:17:50.059887 14206 leader_election.cc:302] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322ebe17500347659bae3076c33d9c5d, a455c7c4dcc14281bf26960d09d2edad; no voters:
I20240502 08:17:50.060261 14602 raft_consensus.cc:2781] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:50.060425 14602 raft_consensus.cc:482] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:50.060541 14602 raft_consensus.cc:3037] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.062676 14602 raft_consensus.cc:504] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
I20240502 08:17:50.063350 14602 leader_election.cc:288] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [CANDIDATE]: Term 1 election: Requested vote from peers 322ebe17500347659bae3076c33d9c5d (127.8.68.194:44407), 6645b5beb31941ffa7658eff74f7be28 (127.8.68.193:41369)
I20240502 08:17:50.063827 14142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f31bf9d6d54b589c5b513238d6b33c" candidate_uuid: "a455c7c4dcc14281bf26960d09d2edad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322ebe17500347659bae3076c33d9c5d"
I20240502 08:17:50.063899 14008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f31bf9d6d54b589c5b513238d6b33c" candidate_uuid: "a455c7c4dcc14281bf26960d09d2edad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6645b5beb31941ffa7658eff74f7be28"
I20240502 08:17:50.064024 14142 raft_consensus.cc:3037] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.064113 14008 raft_consensus.cc:3037] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.066133 14142 raft_consensus.cc:2454] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a455c7c4dcc14281bf26960d09d2edad in term 1.
I20240502 08:17:50.066238 14008 raft_consensus.cc:2454] T 72f31bf9d6d54b589c5b513238d6b33c P 6645b5beb31941ffa7658eff74f7be28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a455c7c4dcc14281bf26960d09d2edad in term 1.
I20240502 08:17:50.066556 14206 leader_election.cc:302] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322ebe17500347659bae3076c33d9c5d, a455c7c4dcc14281bf26960d09d2edad; no voters:
I20240502 08:17:50.066875 14602 raft_consensus.cc:2781] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:50.067334 14602 raft_consensus.cc:686] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [term 1 LEADER]: Becoming Leader. State: Replica: a455c7c4dcc14281bf26960d09d2edad, State: Running, Role: LEADER
I20240502 08:17:50.067883 14602 consensus_queue.cc:237] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } }
W20240502 08:17:50.070992 14188 tablet.cc:2369] T 72f31bf9d6d54b589c5b513238d6b33c P 322ebe17500347659bae3076c33d9c5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:50.072522 13866 catalog_manager.cc:5484] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad reported cstate change: term changed from 0 to 1, leader changed from <none> to a455c7c4dcc14281bf26960d09d2edad (127.8.68.195). New cstate: current_term: 1 leader_uuid: "a455c7c4dcc14281bf26960d09d2edad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 } } peers { permanent_uuid: "a455c7c4dcc14281bf26960d09d2edad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 40509 } } peers { permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 } } }
I20240502 08:17:50.140069 8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:17:50.143069 8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6645b5beb31941ffa7658eff74f7be28 to finish bootstrapping
I20240502 08:17:50.152117 8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 322ebe17500347659bae3076c33d9c5d to finish bootstrapping
I20240502 08:17:50.159725 8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a455c7c4dcc14281bf26960d09d2edad to finish bootstrapping
I20240502 08:17:50.166863 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1655c794693a4259a9bf2901ae7204b1 to finish bootstrapping
I20240502 08:17:50.175843 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3c1c47633c3a4895a150cbd01b5154f0 to finish bootstrapping
I20240502 08:17:50.264706 14447 heartbeater.cc:503] Master 127.8.68.254:43983 was elected leader, sending a full tablet report...
I20240502 08:17:50.492369 14602 consensus_queue.cc:1035] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [LEADER]: Connected to new peer: Peer: permanent_uuid: "322ebe17500347659bae3076c33d9c5d" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:50.500866 14602 consensus_queue.cc:1035] T 72f31bf9d6d54b589c5b513238d6b33c P a455c7c4dcc14281bf26960d09d2edad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645b5beb31941ffa7658eff74f7be28" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:50.662709 14382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.666456 14512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.669963 14122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.671696 13988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.681313 14252 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:50.723392 14577 heartbeater.cc:503] Master 127.8.68.254:43983 was elected leader, sending a full tablet report...
I20240502 08:17:50.842831 14122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.843127 14382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.848475 13988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.851094 14512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:50.852530 14252 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:51.464371 14382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.465950 14122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.468541 14512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.470175 13988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.494092 14252 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:51.663623 14512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.664440 14382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.665766 13988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.670006 14252 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:51.671526 14122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.834882 14122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.835703 14382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.871805 13988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:51.875141 14252 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:51.877532 14512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:52.026083 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 13926
I20240502 08:17:52.027483 14048 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.132843 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 13926
I20240502 08:17:52.152312 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14057
I20240502 08:17:52.153378 14182 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.249976 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14057
I20240502 08:17:52.269850 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14190
I20240502 08:17:52.270843 14312 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.366956 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14190
I20240502 08:17:52.385499 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14320
I20240502 08:17:52.386628 14442 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.481586 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14320
I20240502 08:17:52.499845 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14450
I20240502 08:17:52.500949 14571 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.591023 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14450
I20240502 08:17:52.608359 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 13836
I20240502 08:17:52.609565 13897 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:52.723493 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 13836
2024-05-02T08:17:52Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5945 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5945 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 08:17:52.767563 8467 test_util.cc:268] Using random seed: 1028858419
I20240502 08:17:52.769510 8467 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:52.769588 8467 ts_itest-base.cc:113] --------------
I20240502 08:17:52.769655 8467 ts_itest-base.cc:114] 7 tablet servers
I20240502 08:17:52.769713 8467 ts_itest-base.cc:115] 2 replicas per TS
I20240502 08:17:52.769793 8467 ts_itest-base.cc:116] --------------
2024-05-02T08:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:52Z Disabled control of system clock
I20240502 08:17:52.796826 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:44229
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:44229
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 08:17:53.134567 14803 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:17:53.134902 14803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:53.135008 14803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:53.135123 14803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:53.142746 14803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:53.142880 14803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:53.142943 14803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:53.143011 14803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:53.143106 14803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:53.152465 14803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:44229
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:53.153848 14803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:53.155735 14803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:53.166759 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:53.167024 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:53.167441 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:53.167856 14803 server_base.cc:1034] running on GCE node
I20240502 08:17:53.168718 14803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:53.170235 14803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:53.171550 14803 hybrid_clock.cc:648] HybridClock initialized: now 1714637873171461 us; error 43 us; skew 500 ppm
I20240502 08:17:53.171934 14803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:53.179188 14803 webserver.cc:460] Webserver started at http://127.8.68.254:46339/ using document root <none> and password file <none>
I20240502 08:17:53.180076 14803 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:53.180233 14803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:53.180557 14803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:53.183332 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "481469713cc84d94a634ea9317661c18"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.184123 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "481469713cc84d94a634ea9317661c18"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.189689 14803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:53.193454 14817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:53.194870 14803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:53.195063 14803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "481469713cc84d94a634ea9317661c18"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.195341 14803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:53.212461 14803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:53.213352 14803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:53.213654 14803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:53.232537 14803 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:44229
I20240502 08:17:53.232556 14868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:44229 every 8 connection(s)
I20240502 08:17:53.233951 14803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:53.236892 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14803
I20240502 08:17:53.237134 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:53.238533 14869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.251461 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18: Bootstrap starting.
I20240502 08:17:53.254451 14869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.255502 14869 log.cc:824] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:53.258116 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18: No bootstrap required, opened a new log
I20240502 08:17:53.264379 14869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } }
I20240502 08:17:53.264758 14869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.264875 14869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481469713cc84d94a634ea9317661c18, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.265496 14869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } }
I20240502 08:17:53.265687 14869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:53.265830 14869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:53.265980 14869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.268316 14869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } }
I20240502 08:17:53.268862 14869 leader_election.cc:302] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481469713cc84d94a634ea9317661c18; no voters:
I20240502 08:17:53.269498 14869 leader_election.cc:288] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:53.269661 14874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.270310 14874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [term 1 LEADER]: Becoming Leader. State: Replica: 481469713cc84d94a634ea9317661c18, State: Running, Role: LEADER
I20240502 08:17:53.270913 14874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } }
I20240502 08:17:53.271605 14869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:53.274278 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481469713cc84d94a634ea9317661c18. Latest consensus state: current_term: 1 leader_uuid: "481469713cc84d94a634ea9317661c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } } }
I20240502 08:17:53.274255 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481469713cc84d94a634ea9317661c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481469713cc84d94a634ea9317661c18" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44229 } } }
I20240502 08:17:53.274641 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:53.274639 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:53.278734 14881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:53.284710 14881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:53.294029 14881 catalog_manager.cc:1347] Generated new cluster ID: 7e773eda3c794be1889410425ce244cb
I20240502 08:17:53.294179 14881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:53.323422 14881 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:53.324833 14881 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:53.336577 14881 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 481469713cc84d94a634ea9317661c18: Generated new TSK 0
I20240502 08:17:53.337487 14881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:53.347198 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:17:53.688257 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:53.688575 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:53.688699 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:53.696398 14893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:53.696532 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:53.696691 14893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240502 08:17:53.706377 14893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:53.707859 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:53.709609 14893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:53.721742 14899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:53.721968 14898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:53.722271 14901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:53.722623 14893 server_base.cc:1034] running on GCE node
I20240502 08:17:53.723265 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:53.724246 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:53.725461 14893 hybrid_clock.cc:648] HybridClock initialized: now 1714637873725405 us; error 50 us; skew 500 ppm
I20240502 08:17:53.725786 14893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:53.728161 14893 webserver.cc:460] Webserver started at http://127.8.68.193:38949/ using document root <none> and password file <none>
I20240502 08:17:53.728921 14893 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:53.729040 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:53.729360 14893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:53.732025 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "97037d47bf1249d2b5932b6553b907d4"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.732796 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "97037d47bf1249d2b5932b6553b907d4"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.738094 14893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 08:17:53.741690 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:53.743074 14893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:53.743284 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "97037d47bf1249d2b5932b6553b907d4"
format_stamp: "Formatted at 2024-05-02 08:17:53 on dist-test-slave-2dhs"
I20240502 08:17:53.743526 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:53.773340 14893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:53.774240 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:53.774531 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:53.775594 14893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:53.777185 14893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:53.777299 14893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:53.777416 14893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:53.777489 14893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:53.810835 14893 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:33895
I20240502 08:17:53.810911 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:33895 every 8 connection(s)
I20240502 08:17:53.812141 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:53.819026 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 14893
I20240502 08:17:53.819274 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:53.822651 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:53.825527 15020 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:53.825892 15020 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:53.826844 15020 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:53.829236 14834 ts_manager.cc:194] Registered new tserver with Master: 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:17:53.831521 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:36385
W20240502 08:17:54.163815 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:54.164111 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:54.164222 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:54.171885 15024 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:54.172021 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:54.172148 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240502 08:17:54.181361 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:54.182750 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:54.184551 15024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:54.196286 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:54.196559 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:54.197034 15024 server_base.cc:1034] running on GCE node
W20240502 08:17:54.197245 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:54.197921 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:54.198889 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:54.200079 15024 hybrid_clock.cc:648] HybridClock initialized: now 1714637874200007 us; error 62 us; skew 500 ppm
I20240502 08:17:54.200392 15024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:54.202631 15024 webserver.cc:460] Webserver started at http://127.8.68.194:34967/ using document root <none> and password file <none>
I20240502 08:17:54.203523 15024 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:54.203649 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:54.203992 15024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:54.206596 15024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7fe0e31240d145a28e75262f706ebb89"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.207444 15024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7fe0e31240d145a28e75262f706ebb89"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.212872 15024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 08:17:54.216542 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:54.218024 15024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:54.218209 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7fe0e31240d145a28e75262f706ebb89"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.218436 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:54.232789 15024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:54.233655 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:54.233937 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:54.234961 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:54.236553 15024 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:54.236647 15024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:54.236764 15024 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:54.236845 15024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:54.271741 15024 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:44789
I20240502 08:17:54.271792 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:44789 every 8 connection(s)
I20240502 08:17:54.273207 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:54.275183 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15024
I20240502 08:17:54.275405 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:54.278851 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:54.287135 15151 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:54.287469 15151 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:54.288205 15151 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:54.289532 14834 ts_manager.cc:194] Registered new tserver with Master: 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:17:54.290282 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:51929
W20240502 08:17:54.610061 15154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:54.610381 15154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:54.610515 15154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:54.617995 15154 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:54.618125 15154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:54.618314 15154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240502 08:17:54.627550 15154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:54.628928 15154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:54.630755 15154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:54.642825 15159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:54.642998 15160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:54.643371 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:54.643692 15154 server_base.cc:1034] running on GCE node
I20240502 08:17:54.644357 15154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:54.645360 15154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:54.646592 15154 hybrid_clock.cc:648] HybridClock initialized: now 1714637874646528 us; error 59 us; skew 500 ppm
I20240502 08:17:54.646955 15154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:54.649528 15154 webserver.cc:460] Webserver started at http://127.8.68.195:41101/ using document root <none> and password file <none>
I20240502 08:17:54.650314 15154 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:54.650439 15154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:54.650760 15154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:54.655741 15154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "19d550ab72684bf8aa1382d9c348cd29"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.656550 15154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "19d550ab72684bf8aa1382d9c348cd29"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.661835 15154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:54.665282 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:54.666602 15154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:54.666760 15154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "19d550ab72684bf8aa1382d9c348cd29"
format_stamp: "Formatted at 2024-05-02 08:17:54 on dist-test-slave-2dhs"
I20240502 08:17:54.666966 15154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:54.707140 15154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:54.708134 15154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:54.708436 15154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:54.709427 15154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:54.711019 15154 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:54.711122 15154 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:54.711264 15154 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:54.711339 15154 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 08:17:54.743655 15154 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:43799
I20240502 08:17:54.743724 15280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:43799 every 8 connection(s)
I20240502 08:17:54.745086 15154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:54.749703 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15154
I20240502 08:17:54.749934 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:54.753572 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:54.759677 15281 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:54.760044 15281 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:54.760813 15281 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:54.762388 14834 ts_manager.cc:194] Registered new tserver with Master: 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:17:54.763172 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:60311
I20240502 08:17:54.835561 15020 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:17:55.087571 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:55.087886 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:55.087973 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:55.095547 15284 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:55.095677 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:55.095793 15284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240502 08:17:55.105027 15284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:55.106325 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:55.108064 15284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:55.119828 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:55.119943 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:55.120368 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:55.120628 15284 server_base.cc:1034] running on GCE node
I20240502 08:17:55.121299 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:55.122308 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:55.123512 15284 hybrid_clock.cc:648] HybridClock initialized: now 1714637875123439 us; error 65 us; skew 500 ppm
I20240502 08:17:55.123871 15284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:55.126382 15284 webserver.cc:460] Webserver started at http://127.8.68.196:39553/ using document root <none> and password file <none>
I20240502 08:17:55.127182 15284 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:55.127341 15284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:55.127689 15284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:55.130434 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.131229 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.136670 15284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 08:17:55.140427 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.141907 15284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:55.142077 15284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.142299 15284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:55.164222 15284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:55.165091 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:55.165378 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:55.166290 15284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:55.167832 15284 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:55.167929 15284 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.168048 15284 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:55.168121 15284 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.200511 15284 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:44799
I20240502 08:17:55.200568 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:44799 every 8 connection(s)
I20240502 08:17:55.201846 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:55.206184 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15284
I20240502 08:17:55.206375 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:55.209829 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:55.216325 15411 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:55.216639 15411 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:55.217365 15411 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:55.218853 14834 ts_manager.cc:194] Registered new tserver with Master: 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:17:55.220104 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:40701
I20240502 08:17:55.292695 15151 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:17:55.548604 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:55.548907 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:55.549034 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:55.556540 15414 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:55.556672 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:55.556834 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:17:55.566183 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:55.567596 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:55.569434 15414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:55.581740 15420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:55.582029 15419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:55.582243 15422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:55.582832 15414 server_base.cc:1034] running on GCE node
I20240502 08:17:55.583577 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:55.584579 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:55.585800 15414 hybrid_clock.cc:648] HybridClock initialized: now 1714637875585735 us; error 62 us; skew 500 ppm
I20240502 08:17:55.586133 15414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:55.588586 15414 webserver.cc:460] Webserver started at http://127.8.68.197:46347/ using document root <none> and password file <none>
I20240502 08:17:55.589406 15414 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:55.589531 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:55.589852 15414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:55.592593 15414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.593398 15414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.598853 15414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 08:17:55.602648 15428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.604106 15414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 08:17:55.604336 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:17:55.604540 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:55.622633 15414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:55.623575 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:55.623857 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:55.624891 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:55.626452 15414 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:55.626541 15414 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.626657 15414 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:55.626725 15414 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:55.661573 15414 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:41747
I20240502 08:17:55.661593 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:41747 every 8 connection(s)
I20240502 08:17:55.662856 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:55.672025 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15414
I20240502 08:17:55.672222 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:55.675755 15541 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:55.675886 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:55.676079 15541 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:55.676838 15541 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:55.678174 14834 ts_manager.cc:194] Registered new tserver with Master: 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747)
I20240502 08:17:55.679033 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:44027
I20240502 08:17:55.766350 15281 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:17:56.004714 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:56.005095 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:56.005261 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:56.013016 15544 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:56.013176 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:56.013361 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:17:56.022449 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:56.023901 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:56.025686 15544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:56.040545 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:56.040854 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:56.041054 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:56.041388 15544 server_base.cc:1034] running on GCE node
I20240502 08:17:56.042037 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:56.043025 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:56.044303 15544 hybrid_clock.cc:648] HybridClock initialized: now 1714637876044256 us; error 35 us; skew 500 ppm
I20240502 08:17:56.044672 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:56.047173 15544 webserver.cc:460] Webserver started at http://127.8.68.198:40219/ using document root <none> and password file <none>
I20240502 08:17:56.048039 15544 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:56.048183 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:56.048565 15544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:56.051333 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a8cf9c3ffc34440594482a6718b596e5"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.052143 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a8cf9c3ffc34440594482a6718b596e5"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.057698 15544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:56.061439 15558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.062911 15544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:56.063086 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a8cf9c3ffc34440594482a6718b596e5"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.063330 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:56.090520 15544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:56.091451 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:56.091747 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:56.092729 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:56.094246 15544 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:56.094345 15544 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.094465 15544 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:56.094537 15544 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.127806 15544 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:34645
I20240502 08:17:56.127878 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:34645 every 8 connection(s)
I20240502 08:17:56.129118 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:56.135679 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15544
I20240502 08:17:56.135879 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:17:56.139073 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:0
--local_ip_for_outbound_sockets=127.8.68.199
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:56.142633 15671 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:56.142937 15671 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:56.143668 15671 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:56.145232 14834 ts_manager.cc:194] Registered new tserver with Master: a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645)
I20240502 08:17:56.146039 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:44811
I20240502 08:17:56.222678 15411 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:17:56.475834 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:56.476140 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:56.476269 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:56.483939 15674 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:56.484069 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:56.484222 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:17:56.493397 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:56.494724 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:56.496488 15674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:56.508477 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:56.508574 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:56.508921 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:56.509475 15674 server_base.cc:1034] running on GCE node
I20240502 08:17:56.510072 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:56.511015 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:56.512219 15674 hybrid_clock.cc:648] HybridClock initialized: now 1714637876512132 us; error 85 us; skew 500 ppm
I20240502 08:17:56.512548 15674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:56.514940 15674 webserver.cc:460] Webserver started at http://127.8.68.199:44259/ using document root <none> and password file <none>
I20240502 08:17:56.515769 15674 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:56.515892 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:56.516204 15674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:56.518906 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7757b11bdda3443199e83298ba5c8118"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.519692 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7757b11bdda3443199e83298ba5c8118"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.525023 15674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:17:56.528544 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.529914 15674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:56.530078 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7757b11bdda3443199e83298ba5c8118"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:17:56.530280 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:56.561765 15674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:56.562655 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:56.562963 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:56.563979 15674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:56.565505 15674 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:56.565608 15674 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.565739 15674 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:56.565819 15674 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:56.598022 15674 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:34535
I20240502 08:17:56.598120 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:34535 every 8 connection(s)
I20240502 08:17:56.599432 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:56.609786 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 15674
I20240502 08:17:56.610008 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:17:56.612138 15801 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:17:56.612498 15801 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:56.613227 15801 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:17:56.614573 14833 ts_manager.cc:194] Registered new tserver with Master: 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535)
I20240502 08:17:56.615517 14833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:52763
I20240502 08:17:56.626495 8467 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:17:56.643412 14833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:17:56.681461 15541 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
I20240502 08:17:56.684099 15606 tablet_service.cc:1467] Processing CreateTablet for tablet ed128f6b7a67452db6b11ef0d14937dc (DEFAULT_TABLE table=TestTable [id=f0f068dbe1834794bc2efde74493008c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.685703 15606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed128f6b7a67452db6b11ef0d14937dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.685223 15346 tablet_service.cc:1467] Processing CreateTablet for tablet ed128f6b7a67452db6b11ef0d14937dc (DEFAULT_TABLE table=TestTable [id=f0f068dbe1834794bc2efde74493008c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.686594 15346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed128f6b7a67452db6b11ef0d14937dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.699561 15817 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:17:56.701237 15818 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Bootstrap starting.
I20240502 08:17:56.703423 15817 tablet_bootstrap.cc:654] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.704700 15818 tablet_bootstrap.cc:654] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.704831 15817 log.cc:824] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:56.705910 15818 log.cc:824] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:56.707916 15817 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: No bootstrap required, opened a new log
I20240502 08:17:56.708209 15817 ts_tablet_manager.cc:1367] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 08:17:56.708751 15818 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: No bootstrap required, opened a new log
I20240502 08:17:56.709021 15818 ts_tablet_manager.cc:1367] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:17:56.717362 15817 raft_consensus.cc:348] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:56.717562 15818 raft_consensus.cc:348] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:56.717849 15817 raft_consensus.cc:374] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.717913 15818 raft_consensus.cc:374] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.717964 15817 raft_consensus.cc:729] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.718025 15818 raft_consensus.cc:729] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.718719 15817 consensus_queue.cc:260] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:56.718717 15818 consensus_queue.cc:260] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:56.720567 15818 ts_tablet_manager.cc:1398] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 08:17:56.720592 15817 ts_tablet_manager.cc:1398] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20240502 08:17:56.720968 15671 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:17:56.881150 15672 tablet.cc:2369] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:56.954628 15412 tablet.cc:2369] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:57.075115 15822 raft_consensus.cc:482] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.075425 15822 raft_consensus.cc:504] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:57.076984 15822 leader_election.cc:288] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:17:57.083807 15366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed128f6b7a67452db6b11ef0d14937dc" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" is_pre_election: true
I20240502 08:17:57.084328 15366 raft_consensus.cc:2454] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 0.
I20240502 08:17:57.084986 15559 leader_election.cc:302] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 83659c8dbbcd4142abdbcb243eeec8d4, a8cf9c3ffc34440594482a6718b596e5; no voters:
I20240502 08:17:57.085417 15822 raft_consensus.cc:2781] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.085561 15822 raft_consensus.cc:482] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.085652 15822 raft_consensus.cc:3037] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.087680 15822 raft_consensus.cc:504] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:57.088225 15822 leader_election.cc:288] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 1 election: Requested vote from peers 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:17:57.088860 15366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed128f6b7a67452db6b11ef0d14937dc" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
I20240502 08:17:57.089046 15366 raft_consensus.cc:3037] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.091307 15366 raft_consensus.cc:2454] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 1.
I20240502 08:17:57.091739 15559 leader_election.cc:302] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 83659c8dbbcd4142abdbcb243eeec8d4, a8cf9c3ffc34440594482a6718b596e5; no voters:
I20240502 08:17:57.092041 15822 raft_consensus.cc:2781] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.092495 15822 raft_consensus.cc:686] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 LEADER]: Becoming Leader. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Running, Role: LEADER
I20240502 08:17:57.093009 15822 consensus_queue.cc:237] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:17:57.096856 14834 catalog_manager.cc:5484] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "a8cf9c3ffc34440594482a6718b596e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:17:57.126824 8467 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:17:57.129393 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 97037d47bf1249d2b5932b6553b907d4 to finish bootstrapping
I20240502 08:17:57.140514 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7fe0e31240d145a28e75262f706ebb89 to finish bootstrapping
I20240502 08:17:57.149749 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 19d550ab72684bf8aa1382d9c348cd29 to finish bootstrapping
I20240502 08:17:57.158812 8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 83659c8dbbcd4142abdbcb243eeec8d4 to finish bootstrapping
I20240502 08:17:57.166067 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 382fd0c6daaf4eddb1c57557e9b716d1 to finish bootstrapping
I20240502 08:17:57.175199 8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a8cf9c3ffc34440594482a6718b596e5 to finish bootstrapping
I20240502 08:17:57.182729 8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7757b11bdda3443199e83298ba5c8118 to finish bootstrapping
I20240502 08:17:57.191318 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15284
I20240502 08:17:57.192328 15406 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.279548 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15284
I20240502 08:17:57.296360 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15414
I20240502 08:17:57.297601 15536 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.379287 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15414
I20240502 08:17:57.395232 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15544
I20240502 08:17:57.396333 15666 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.485702 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15544
I20240502 08:17:57.502101 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15674
I20240502 08:17:57.503432 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.582898 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15674
I20240502 08:18:01.350414 14834 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:01.373560 14955 tablet_service.cc:1467] Processing CreateTablet for tablet b2f290f16fe044658ed9b6b9a0a71f08 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.373940 15216 tablet_service.cc:1467] Processing CreateTablet for tablet b2f290f16fe044658ed9b6b9a0a71f08 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.374380 15215 tablet_service.cc:1467] Processing CreateTablet for tablet f9acb74154404bd8829cc19db214068a (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.375509 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f290f16fe044658ed9b6b9a0a71f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.375589 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9acb74154404bd8829cc19db214068a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.376188 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f290f16fe044658ed9b6b9a0a71f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.373533 14954 tablet_service.cc:1467] Processing CreateTablet for tablet cadca3ac819042f0bee9f3251283a6f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.378126 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadca3ac819042f0bee9f3251283a6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.379004 15085 tablet_service.cc:1467] Processing CreateTablet for tablet f9acb74154404bd8829cc19db214068a (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.380492 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9acb74154404bd8829cc19db214068a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.383364 15086 tablet_service.cc:1467] Processing CreateTablet for tablet cadca3ac819042f0bee9f3251283a6f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.383881 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadca3ac819042f0bee9f3251283a6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.393853 15847 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:01.395973 15846 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:01.398157 15847 tablet_bootstrap.cc:654] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.400331 15846 tablet_bootstrap.cc:654] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.400799 15847 log.cc:824] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:01.401768 15846 log.cc:824] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:01.410925 15847 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:01.411412 15847 ts_tablet_manager.cc:1367] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240502 08:18:01.413162 15850 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:01.414232 15846 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:01.414544 15846 ts_tablet_manager.cc:1367] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.011s
I20240502 08:18:01.416096 15850 tablet_bootstrap.cc:654] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.417110 15850 log.cc:824] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:01.420409 15850 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:01.420683 15850 ts_tablet_manager.cc:1367] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:18:01.420601 15847 raft_consensus.cc:348] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.421072 15847 raft_consensus.cc:374] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.421182 15847 raft_consensus.cc:729] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.421950 15847 consensus_queue.cc:260] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.423898 15846 raft_consensus.cc:348] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.424417 15846 raft_consensus.cc:374] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.424610 15846 raft_consensus.cc:729] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.425438 15847 ts_tablet_manager.cc:1398] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.014s user 0.010s sys 0.001s
I20240502 08:18:01.425662 15846 consensus_queue.cc:260] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.425961 15847 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:01.428812 15847 tablet_bootstrap.cc:654] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.428978 15850 raft_consensus.cc:348] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.429347 15850 raft_consensus.cc:374] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.429452 15850 raft_consensus.cc:729] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.430148 15850 consensus_queue.cc:260] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.431425 15847 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:01.431641 15847 ts_tablet_manager.cc:1367] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:18:01.431859 15846 ts_tablet_manager.cc:1398] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20240502 08:18:01.433470 15850 ts_tablet_manager.cc:1398] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20240502 08:18:01.433705 15847 raft_consensus.cc:348] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.433876 15847 raft_consensus.cc:374] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.433894 15850 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:01.434015 15847 raft_consensus.cc:729] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.434253 15847 consensus_queue.cc:260] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.434631 15847 ts_tablet_manager.cc:1398] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:01.435593 15850 tablet_bootstrap.cc:654] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.436018 15846 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:01.437700 15846 tablet_bootstrap.cc:654] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.442410 15846 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:01.442616 15846 ts_tablet_manager.cc:1367] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 08:18:01.443646 15846 raft_consensus.cc:348] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.443804 15846 raft_consensus.cc:374] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.443857 15850 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:01.443869 15846 raft_consensus.cc:729] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.444041 15850 ts_tablet_manager.cc:1367] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 08:18:01.444096 15846 consensus_queue.cc:260] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.444533 15846 ts_tablet_manager.cc:1398] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:01.445451 15850 raft_consensus.cc:348] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.445675 15850 raft_consensus.cc:374] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.445775 15850 raft_consensus.cc:729] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.446061 15850 consensus_queue.cc:260] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.446549 15850 ts_tablet_manager.cc:1398] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:01.449872 15853 raft_consensus.cc:482] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:01.450094 15853 raft_consensus.cc:504] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.451915 15853 leader_election.cc:288] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:01.457691 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadca3ac819042f0bee9f3251283a6f3" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:01.458180 15106 raft_consensus.cc:2454] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:01.458827 14911 leader_election.cc:302] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:01.459220 15853 raft_consensus.cc:2781] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:01.459403 15853 raft_consensus.cc:482] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:01.459512 15853 raft_consensus.cc:3037] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.461535 15853 raft_consensus.cc:504] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.462042 15853 leader_election.cc:288] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:01.462525 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadca3ac819042f0bee9f3251283a6f3" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:01.462704 15106 raft_consensus.cc:3037] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.464820 15106 raft_consensus.cc:2454] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:01.465234 14911 leader_election.cc:302] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:01.465521 15853 raft_consensus.cc:2781] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:01.465939 15853 raft_consensus.cc:686] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:01.466429 15853 consensus_queue.cc:237] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:01.470199 14833 catalog_manager.cc:5484] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
W20240502 08:18:01.501457 15282 tablet.cc:2369] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:01.528219 15855 raft_consensus.cc:482] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:01.528455 15855 raft_consensus.cc:504] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.529930 15855 leader_election.cc:288] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
W20240502 08:18:01.530212 15152 tablet.cc:2369] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:01.536384 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9acb74154404bd8829cc19db214068a" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:01.536870 15236 raft_consensus.cc:2454] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 0.
I20240502 08:18:01.537453 15041 leader_election.cc:302] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:01.537833 15855 raft_consensus.cc:2781] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:01.537942 15855 raft_consensus.cc:482] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:01.538033 15855 raft_consensus.cc:3037] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.540280 15855 raft_consensus.cc:504] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.540875 15855 leader_election.cc:288] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:01.541368 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9acb74154404bd8829cc19db214068a" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:01.541550 15236 raft_consensus.cc:3037] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.543690 15236 raft_consensus.cc:2454] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 1.
I20240502 08:18:01.544193 15041 leader_election.cc:302] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:01.544538 15855 raft_consensus.cc:2781] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:01.545085 15855 raft_consensus.cc:686] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:18:01.545605 15855 consensus_queue.cc:237] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.549960 14833 catalog_manager.cc:5484] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } }
W20240502 08:18:01.569957 15021 tablet.cc:2369] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:01.736907 15852 raft_consensus.cc:482] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:01.737107 15852 raft_consensus.cc:504] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.738652 15852 leader_election.cc:288] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:01.744936 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4" is_pre_election: true
I20240502 08:18:01.745390 14975 raft_consensus.cc:2454] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 0.
I20240502 08:18:01.746043 15172 leader_election.cc:302] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:01.746480 15852 raft_consensus.cc:2781] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:01.746613 15852 raft_consensus.cc:482] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:01.746721 15852 raft_consensus.cc:3037] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.748960 15852 raft_consensus.cc:504] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.749495 15852 leader_election.cc:288] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:01.750092 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:18:01.750298 14975 raft_consensus.cc:3037] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.753427 14975 raft_consensus.cc:2454] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 1.
I20240502 08:18:01.753897 15172 leader_election.cc:302] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:01.754313 15852 raft_consensus.cc:2781] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:01.754879 15852 raft_consensus.cc:686] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:01.755458 15852 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:01.759824 14833 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } }
I20240502 08:18:01.870395 15853 consensus_queue.cc:1035] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.037966 15855 consensus_queue.cc:1035] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.173445 15852 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:02.181142 15867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:02.181473 15867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:02.189114 15867 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 199.34 ms
time per row: 0.0996702 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:18:03.460566 14832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:03.470858 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 9738c0a921014b7d9b8c9a1be4cbb433 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.471458 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9738c0a921014b7d9b8c9a1be4cbb433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.471784 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 8e685b7cf44341c5b9e89360d8488338 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.472023 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 8e685b7cf44341c5b9e89360d8488338 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.472216 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e685b7cf44341c5b9e89360d8488338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.471113 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 9738c0a921014b7d9b8c9a1be4cbb433 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.472412 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e685b7cf44341c5b9e89360d8488338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.472587 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9738c0a921014b7d9b8c9a1be4cbb433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.472890 15215 tablet_service.cc:1467] Processing CreateTablet for tablet df91c5f0a46c43d78f8f7aeef7281e14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.473223 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df91c5f0a46c43d78f8f7aeef7281e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.479347 15085 tablet_service.cc:1467] Processing CreateTablet for tablet df91c5f0a46c43d78f8f7aeef7281e14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.479948 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df91c5f0a46c43d78f8f7aeef7281e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.480628 15931 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:03.482717 15931 tablet_bootstrap.cc:654] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.487987 15930 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:03.490525 15930 tablet_bootstrap.cc:654] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.492380 15931 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:03.492656 15931 ts_tablet_manager.cc:1367] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:18:03.493578 15931 raft_consensus.cc:348] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.493978 15931 raft_consensus.cc:374] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.494091 15931 raft_consensus.cc:729] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.494366 15932 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:03.494524 15931 consensus_queue.cc:260] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.495314 15931 ts_tablet_manager.cc:1398] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:03.495981 15932 tablet_bootstrap.cc:654] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.496008 15931 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:03.498076 15931 tablet_bootstrap.cc:654] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.498168 15930 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:03.498400 15930 ts_tablet_manager.cc:1367] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20240502 08:18:03.499383 15930 raft_consensus.cc:348] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.499747 15930 raft_consensus.cc:374] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.499897 15930 raft_consensus.cc:729] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.500272 15930 consensus_queue.cc:260] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.500926 15930 ts_tablet_manager.cc:1398] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:03.501221 15930 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:03.502584 15932 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:03.502918 15932 ts_tablet_manager.cc:1367] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:18:03.503427 15930 tablet_bootstrap.cc:654] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.503904 15932 raft_consensus.cc:348] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.504124 15932 raft_consensus.cc:374] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.504233 15932 raft_consensus.cc:729] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.504976 15932 consensus_queue.cc:260] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.505739 15932 ts_tablet_manager.cc:1398] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:03.506260 15932 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:03.508193 15931 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:03.508221 15932 tablet_bootstrap.cc:654] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.508524 15931 ts_tablet_manager.cc:1367] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240502 08:18:03.510171 15931 raft_consensus.cc:348] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.510502 15931 raft_consensus.cc:374] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.510603 15931 raft_consensus.cc:729] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.510932 15931 consensus_queue.cc:260] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.511749 15931 ts_tablet_manager.cc:1398] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:03.512723 15930 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:03.512929 15930 ts_tablet_manager.cc:1367] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 08:18:03.513722 15930 raft_consensus.cc:348] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.513903 15930 raft_consensus.cc:374] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.513988 15930 raft_consensus.cc:729] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.514230 15930 consensus_queue.cc:260] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.514870 15930 ts_tablet_manager.cc:1398] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:03.516412 15932 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:03.516657 15932 ts_tablet_manager.cc:1367] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 08:18:03.517493 15932 raft_consensus.cc:348] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.517674 15932 raft_consensus.cc:374] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.517750 15932 raft_consensus.cc:729] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.517973 15932 consensus_queue.cc:260] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.518522 15932 ts_tablet_manager.cc:1398] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:03.520803 15929 raft_consensus.cc:482] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.520961 15929 raft_consensus.cc:504] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.521543 15929 leader_election.cc:288] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:03.522046 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9738c0a921014b7d9b8c9a1be4cbb433" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:03.522315 15106 raft_consensus.cc:2454] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:03.522749 14911 leader_election.cc:302] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:03.523041 15929 raft_consensus.cc:2781] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.523141 15929 raft_consensus.cc:482] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.523231 15929 raft_consensus.cc:3037] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.525216 15929 raft_consensus.cc:504] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.525624 15929 leader_election.cc:288] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:03.526171 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9738c0a921014b7d9b8c9a1be4cbb433" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:03.526388 15106 raft_consensus.cc:3037] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.528363 15106 raft_consensus.cc:2454] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:03.528859 14911 leader_election.cc:302] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:03.529191 15929 raft_consensus.cc:2781] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.529330 15929 raft_consensus.cc:686] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:03.529600 15929 consensus_queue.cc:237] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.532198 14834 catalog_manager.cc:5484] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
I20240502 08:18:03.731017 15929 raft_consensus.cc:482] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.731302 15923 raft_consensus.cc:482] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.731513 15923 raft_consensus.cc:504] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.731633 15929 raft_consensus.cc:504] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.732112 15923 leader_election.cc:288] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:03.732272 15929 leader_election.cc:288] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:03.732787 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e685b7cf44341c5b9e89360d8488338" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4" is_pre_election: true
I20240502 08:18:03.732924 15923 raft_consensus.cc:482] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.733081 14975 raft_consensus.cc:2454] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 0.
I20240502 08:18:03.733093 15923 raft_consensus.cc:504] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.733764 15923 leader_election.cc:288] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:03.733793 15172 leader_election.cc:302] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:03.734081 15923 raft_consensus.cc:2781] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.734169 15923 raft_consensus.cc:482] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.734290 15923 raft_consensus.cc:3037] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.738222 15923 raft_consensus.cc:504] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.739107 15923 leader_election.cc:288] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:03.740383 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e685b7cf44341c5b9e89360d8488338" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:18:03.740676 14975 raft_consensus.cc:3037] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.742285 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e685b7cf44341c5b9e89360d8488338" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:03.742590 15236 raft_consensus.cc:2379] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97037d47bf1249d2b5932b6553b907d4 in current term 1: Already voted for candidate 19d550ab72684bf8aa1382d9c348cd29 in this term.
I20240502 08:18:03.743124 14975 raft_consensus.cc:2454] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 1.
I20240502 08:18:03.743186 14910 leader_election.cc:302] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 97037d47bf1249d2b5932b6553b907d4; no voters: 19d550ab72684bf8aa1382d9c348cd29
I20240502 08:18:03.743521 15929 raft_consensus.cc:2726] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:03.743619 15172 leader_election.cc:302] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:03.743918 15923 raft_consensus.cc:2781] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.744495 15923 raft_consensus.cc:686] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:03.744783 15923 consensus_queue.cc:237] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:03.747582 14834 catalog_manager.cc:5484] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
I20240502 08:18:03.751757 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:03.751962 15106 raft_consensus.cc:2454] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 0.
I20240502 08:18:03.752337 15172 leader_election.cc:302] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:03.752683 15923 raft_consensus.cc:2781] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.752785 15923 raft_consensus.cc:482] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.752933 15923 raft_consensus.cc:3037] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.754890 15923 raft_consensus.cc:504] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.755409 15923 leader_election.cc:288] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:03.755934 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:03.756145 15106 raft_consensus.cc:3037] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.758020 15106 raft_consensus.cc:2454] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 1.
I20240502 08:18:03.758394 15172 leader_election.cc:302] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:03.758641 15923 raft_consensus.cc:2781] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.758801 15923 raft_consensus.cc:686] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:03.759040 15923 consensus_queue.cc:237] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:03.761340 14834 catalog_manager.cc:5484] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } }
I20240502 08:18:04.104951 15929 consensus_queue.cc:1035] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:04.116387 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:04.116731 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:04.124471 15948 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:18:04.270259 15106 raft_consensus.cc:1261] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.271636 15923 consensus_queue.cc:1035] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.285881 14975 raft_consensus.cc:1261] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.300343 15923 consensus_queue.cc:1035] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.342499 15980 mvcc.cc:205] Tried to move back new op lower bound from 7023156773946789888 to 7023156771878440960. Current Snapshot: MvccSnapshot[applied={T|T < 7023156773946789888}]
I20240502 08:18:04.371156 15989 mvcc.cc:205] Tried to move back new op lower bound from 7023156773997084672 to 7023156771820425216. Current Snapshot: MvccSnapshot[applied={T|T < 7023156773997084672}]
INSERT report
rows total: 2000
time total: 195.936 ms
time per row: 0.0979679 ms
I20240502 08:18:04.474830 14829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:04.483714 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 68b0f3cfa17644449993f63affcbe2be (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.483776 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 68b0f3cfa17644449993f63affcbe2be (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.484205 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b0f3cfa17644449993f63affcbe2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.484215 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b0f3cfa17644449993f63affcbe2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.484208 15215 tablet_service.cc:1467] Processing CreateTablet for tablet 4485a1686beb45d7ad56010d2a7c9fc9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.484416 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ca0fbedbe43e2a42fd849c233ded3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.484620 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4485a1686beb45d7ad56010d2a7c9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.485136 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca0fbedbe43e2a42fd849c233ded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.491402 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 4485a1686beb45d7ad56010d2a7c9fc9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.491834 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4485a1686beb45d7ad56010d2a7c9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.492605 16003 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:04.493714 16005 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:04.495154 16003 tablet_bootstrap.cc:654] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.495522 16005 tablet_bootstrap.cc:654] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.495709 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ca0fbedbe43e2a42fd849c233ded3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.496171 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca0fbedbe43e2a42fd849c233ded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.500808 16003 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:04.500965 16003 ts_tablet_manager.cc:1367] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:18:04.501745 16003 raft_consensus.cc:348] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.502753 16003 raft_consensus.cc:374] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.503049 16003 raft_consensus.cc:729] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.503345 16003 consensus_queue.cc:260] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.503811 16004 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:04.504209 16003 ts_tablet_manager.cc:1398] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:04.504475 16003 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:04.504679 16005 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:04.504865 16005 ts_tablet_manager.cc:1367] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 08:18:04.505609 16004 tablet_bootstrap.cc:654] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.505692 16005 raft_consensus.cc:348] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.505954 16005 raft_consensus.cc:374] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.506098 16005 raft_consensus.cc:729] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.506165 16003 tablet_bootstrap.cc:654] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.506443 16005 consensus_queue.cc:260] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.508065 16005 ts_tablet_manager.cc:1398] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:04.508369 16005 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:04.511870 16005 tablet_bootstrap.cc:654] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.514336 16003 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:04.514524 16003 ts_tablet_manager.cc:1367] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240502 08:18:04.514554 16005 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:04.514691 16005 ts_tablet_manager.cc:1367] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:04.515395 16003 raft_consensus.cc:348] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.515447 16005 raft_consensus.cc:348] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.515627 16005 raft_consensus.cc:374] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.515678 16003 raft_consensus.cc:374] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.515720 16005 raft_consensus.cc:729] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.515750 16003 raft_consensus.cc:729] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.516012 16004 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:04.516010 16005 consensus_queue.cc:260] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.516010 16003 consensus_queue.cc:260] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.516162 16004 ts_tablet_manager.cc:1367] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 08:18:04.516572 16005 ts_tablet_manager.cc:1398] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.516620 16003 ts_tablet_manager.cc:1398] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:04.516952 16004 raft_consensus.cc:348] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.517158 16004 raft_consensus.cc:374] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.517251 16004 raft_consensus.cc:729] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.517455 16004 consensus_queue.cc:260] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.517941 16004 ts_tablet_manager.cc:1398] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:04.518258 16004 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:04.519892 16004 tablet_bootstrap.cc:654] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.521884 16004 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:04.522040 16004 ts_tablet_manager.cc:1367] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:04.522635 16004 raft_consensus.cc:348] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.522889 16004 raft_consensus.cc:374] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.523007 16004 raft_consensus.cc:729] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.523329 16004 consensus_queue.cc:260] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.523937 16004 ts_tablet_manager.cc:1398] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.551342 15923 raft_consensus.cc:482] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.551574 15923 raft_consensus.cc:504] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.552173 15923 leader_election.cc:288] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:04.552788 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ca0fbedbe43e2a42fd849c233ded3" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:04.552989 15106 raft_consensus.cc:2454] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 0.
I20240502 08:18:04.553395 15172 leader_election.cc:302] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:04.553679 15923 raft_consensus.cc:2781] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.553766 15923 raft_consensus.cc:482] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.553841 15923 raft_consensus.cc:3037] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.555703 15923 raft_consensus.cc:504] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.556140 15923 leader_election.cc:288] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:04.556684 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7ca0fbedbe43e2a42fd849c233ded3" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:04.556852 15106 raft_consensus.cc:3037] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.558708 15106 raft_consensus.cc:2454] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 1.
I20240502 08:18:04.559111 15172 leader_election.cc:302] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:04.559417 15923 raft_consensus.cc:2781] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.559595 15923 raft_consensus.cc:686] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:04.559813 15923 consensus_queue.cc:237] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.562238 14831 catalog_manager.cc:5484] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } }
I20240502 08:18:04.582401 15985 raft_consensus.cc:482] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.582577 15985 raft_consensus.cc:504] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.583173 15985 leader_election.cc:288] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:04.583779 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:04.584022 15236 raft_consensus.cc:2454] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:04.584439 14910 leader_election.cc:302] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:04.584723 15985 raft_consensus.cc:2781] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.584834 15985 raft_consensus.cc:482] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.584921 15985 raft_consensus.cc:3037] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.586833 15985 raft_consensus.cc:504] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.587338 15985 leader_election.cc:288] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:04.587889 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:04.588101 15236 raft_consensus.cc:3037] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.589951 15236 raft_consensus.cc:2454] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:04.590368 14910 leader_election.cc:302] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:04.590677 15985 raft_consensus.cc:2781] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.590813 15985 raft_consensus.cc:686] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:04.591050 15985 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:04.593670 14831 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
I20240502 08:18:04.641117 15985 raft_consensus.cc:482] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.641379 15985 raft_consensus.cc:504] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.641911 15985 leader_election.cc:288] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:04.642546 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:04.642755 15106 raft_consensus.cc:2454] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:04.643169 14911 leader_election.cc:302] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:04.643501 15985 raft_consensus.cc:2781] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.643595 15985 raft_consensus.cc:482] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.643667 15985 raft_consensus.cc:3037] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.645537 15985 raft_consensus.cc:504] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.646013 15985 leader_election.cc:288] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:04.646584 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:04.646768 15106 raft_consensus.cc:3037] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.648751 15106 raft_consensus.cc:2454] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:04.649169 14911 leader_election.cc:302] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:04.649489 15985 raft_consensus.cc:2781] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.649641 15985 raft_consensus.cc:686] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:04.649857 15985 consensus_queue.cc:237] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:04.652565 14831 catalog_manager.cc:5484] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } }
W20240502 08:18:05.032359 16015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:05.032747 16015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:05.040625 16015 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:18:05.110241 15923 consensus_queue.cc:1035] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.125749 15985 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.192119 15985 consensus_queue.cc:1035] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 198.263 ms
time per row: 0.0991314 ms
I20240502 08:18:05.383090 14825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:05.391620 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 4c94e28f8f2b47cba77ed459c8e731d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.391800 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 4c94e28f8f2b47cba77ed459c8e731d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.392094 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c94e28f8f2b47cba77ed459c8e731d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.391992 15215 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc254d1803141fbab91423631ab8855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.392175 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c94e28f8f2b47cba77ed459c8e731d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.392375 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc254d1803141fbab91423631ab8855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.395905 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 73acd88beabf42679643daf4e62dea3b (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.395915 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc254d1803141fbab91423631ab8855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.396107 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 73acd88beabf42679643daf4e62dea3b (DEFAULT_TABLE table=rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.396282 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc254d1803141fbab91423631ab8855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.396289 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73acd88beabf42679643daf4e62dea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.396472 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73acd88beabf42679643daf4e62dea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.409260 16071 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:05.409376 16072 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:05.410298 16070 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:05.411299 16071 tablet_bootstrap.cc:654] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.411832 16072 tablet_bootstrap.cc:654] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.412127 16070 tablet_bootstrap.cc:654] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.413511 16071 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:05.413682 16071 ts_tablet_manager.cc:1367] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:18:05.414028 16070 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:05.414191 16070 ts_tablet_manager.cc:1367] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:18:05.414252 16072 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:05.414458 16072 ts_tablet_manager.cc:1367] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:05.414503 16071 raft_consensus.cc:348] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.414697 16071 raft_consensus.cc:374] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.414780 16071 raft_consensus.cc:729] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.414939 16070 raft_consensus.cc:348] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.415015 16071 consensus_queue.cc:260] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.415158 16070 raft_consensus.cc:374] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.415238 16070 raft_consensus.cc:729] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.415189 16072 raft_consensus.cc:348] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.415417 16072 raft_consensus.cc:374] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.415511 16072 raft_consensus.cc:729] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.415596 16071 ts_tablet_manager.cc:1398] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.415472 16070 consensus_queue.cc:260] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.415729 16072 consensus_queue.cc:260] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.415879 16071 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:05.416292 16072 ts_tablet_manager.cc:1398] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.416536 16072 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:05.416646 16070 ts_tablet_manager.cc:1398] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:05.416931 16070 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:05.417585 16071 tablet_bootstrap.cc:654] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.418547 16070 tablet_bootstrap.cc:654] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.418982 16072 tablet_bootstrap.cc:654] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.429345 16070 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:05.429529 16070 ts_tablet_manager.cc:1367] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240502 08:18:05.429562 16072 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:05.429736 16071 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:05.429733 16072 ts_tablet_manager.cc:1367] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 08:18:05.429904 16071 ts_tablet_manager.cc:1367] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20240502 08:18:05.430271 16070 raft_consensus.cc:348] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.430454 16070 raft_consensus.cc:374] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.430537 16070 raft_consensus.cc:729] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.430629 16072 raft_consensus.cc:348] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.430735 16071 raft_consensus.cc:348] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.430864 16072 raft_consensus.cc:374] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.430936 16071 raft_consensus.cc:374] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.430852 16070 consensus_queue.cc:260] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.430961 16072 raft_consensus.cc:729] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.431016 16071 raft_consensus.cc:729] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.431334 16072 consensus_queue.cc:260] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.431434 16070 ts_tablet_manager.cc:1398] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.431346 16071 consensus_queue.cc:260] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.431882 16071 ts_tablet_manager.cc:1398] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.432498 16072 ts_tablet_manager.cc:1398] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:05.443037 16063 raft_consensus.cc:482] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.443274 16063 raft_consensus.cc:504] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.443929 16063 leader_election.cc:288] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:05.444324 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c94e28f8f2b47cba77ed459c8e731d0" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89" is_pre_election: true
I20240502 08:18:05.444574 15106 raft_consensus.cc:2454] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:05.444955 14911 leader_election.cc:302] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:05.445206 16063 raft_consensus.cc:2781] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.445317 16063 raft_consensus.cc:482] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.445384 16063 raft_consensus.cc:3037] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.447505 16063 raft_consensus.cc:504] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.447914 16063 leader_election.cc:288] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:05.448395 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c94e28f8f2b47cba77ed459c8e731d0" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:05.448596 15106 raft_consensus.cc:3037] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.450438 15106 raft_consensus.cc:2454] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:05.450820 14911 leader_election.cc:302] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:05.451102 16063 raft_consensus.cc:2781] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.451233 16063 raft_consensus.cc:686] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:05.451498 16063 consensus_queue.cc:237] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:05.453930 14829 catalog_manager.cc:5484] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } }
I20240502 08:18:05.508244 15924 raft_consensus.cc:482] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.508455 15924 raft_consensus.cc:504] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.508936 15924 leader_election.cc:288] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:05.509500 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73acd88beabf42679643daf4e62dea3b" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:05.509712 15236 raft_consensus.cc:2454] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 0.
I20240502 08:18:05.510195 15041 leader_election.cc:302] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:05.510490 15924 raft_consensus.cc:2781] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.510591 15924 raft_consensus.cc:482] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.510682 15924 raft_consensus.cc:3037] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.512676 15924 raft_consensus.cc:504] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.513098 15924 leader_election.cc:288] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:05.513644 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73acd88beabf42679643daf4e62dea3b" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:05.513830 15236 raft_consensus.cc:3037] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.516223 15236 raft_consensus.cc:2454] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 1.
I20240502 08:18:05.516598 15041 leader_election.cc:302] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:05.516888 15924 raft_consensus.cc:2781] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.517300 15924 raft_consensus.cc:686] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:18:05.517601 15924 consensus_queue.cc:237] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:05.520112 14829 catalog_manager.cc:5484] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } }
I20240502 08:18:05.784715 16063 raft_consensus.cc:482] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.784940 16063 raft_consensus.cc:504] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.785456 16063 leader_election.cc:288] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:05.786103 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc254d1803141fbab91423631ab8855" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:05.786394 15236 raft_consensus.cc:2454] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:05.786828 14910 leader_election.cc:302] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:05.787117 16063 raft_consensus.cc:2781] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.787205 16063 raft_consensus.cc:482] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.787317 16063 raft_consensus.cc:3037] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.789109 16063 raft_consensus.cc:504] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.789605 16063 leader_election.cc:288] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:05.790143 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc254d1803141fbab91423631ab8855" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:05.790349 15236 raft_consensus.cc:3037] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.792235 15236 raft_consensus.cc:2454] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:05.792716 14910 leader_election.cc:302] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:05.793015 16063 raft_consensus.cc:2781] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.793179 16063 raft_consensus.cc:686] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:05.793503 16063 consensus_queue.cc:237] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:05.795763 14829 catalog_manager.cc:5484] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
I20240502 08:18:05.907032 15924 consensus_queue.cc:1035] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.969718 16063 consensus_queue.cc:1035] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:06.214388 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:06.214753 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:06.222456 16083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 08:18:06.373541 15235 raft_consensus.cc:1261] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.374270 15953 consensus_queue.cc:1035] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 187.379 ms
time per row: 0.0936895 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:18:07.489933 14828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59858:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:07.498723 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2e9479648049ca94e7e5bfeb98db6e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.498939 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2e9479648049ca94e7e5bfeb98db6e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.499187 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2e9479648049ca94e7e5bfeb98db6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.499368 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2e9479648049ca94e7e5bfeb98db6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.500138 15215 tablet_service.cc:1467] Processing CreateTablet for tablet 445454fd0e3d405d87fc7ef20929e862 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.500589 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445454fd0e3d405d87fc7ef20929e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.500710 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa2b2b96e134484a50edaac02dcc2a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.501034 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa2b2b96e134484a50edaac02dcc2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.503620 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 445454fd0e3d405d87fc7ef20929e862 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.503995 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445454fd0e3d405d87fc7ef20929e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.505983 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa2b2b96e134484a50edaac02dcc2a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.506361 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa2b2b96e134484a50edaac02dcc2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.513691 16141 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:07.515139 16142 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:07.515830 16141 tablet_bootstrap.cc:654] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.517664 16142 tablet_bootstrap.cc:654] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.520073 16143 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:07.523002 16143 tablet_bootstrap.cc:654] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.533370 16141 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:07.533418 16142 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:07.533566 16142 ts_tablet_manager.cc:1367] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240502 08:18:07.533564 16141 ts_tablet_manager.cc:1367] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 08:18:07.533891 16143 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:07.534058 16143 ts_tablet_manager.cc:1367] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 08:18:07.534273 16142 raft_consensus.cc:348] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.534457 16142 raft_consensus.cc:374] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.534348 16141 raft_consensus.cc:348] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.534546 16141 raft_consensus.cc:374] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.534597 16142 raft_consensus.cc:729] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.534651 16141 raft_consensus.cc:729] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.534855 16142 consensus_queue.cc:260] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.534881 16141 consensus_queue.cc:260] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.534901 16143 raft_consensus.cc:348] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.535127 16143 raft_consensus.cc:374] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.535609 16143 raft_consensus.cc:729] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.536195 16142 ts_tablet_manager.cc:1398] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:07.536465 16142 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29: Bootstrap starting.
I20240502 08:18:07.536460 16143 consensus_queue.cc:260] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.537328 16143 ts_tablet_manager.cc:1398] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:18:07.538152 16142 tablet_bootstrap.cc:654] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.539574 16143 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: Bootstrap starting.
I20240502 08:18:07.542651 16142 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29: No bootstrap required, opened a new log
I20240502 08:18:07.542809 16142 ts_tablet_manager.cc:1367] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:18:07.542886 16143 tablet_bootstrap.cc:654] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.543505 16142 raft_consensus.cc:348] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.543730 16142 raft_consensus.cc:374] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.543816 16142 raft_consensus.cc:729] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.543825 16141 ts_tablet_manager.cc:1398] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20240502 08:18:07.544066 16142 consensus_queue.cc:260] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.544207 16141 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Bootstrap starting.
I20240502 08:18:07.545925 16143 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: No bootstrap required, opened a new log
I20240502 08:18:07.546006 16142 ts_tablet_manager.cc:1398] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:07.546900 16141 tablet_bootstrap.cc:654] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.546963 16143 ts_tablet_manager.cc:1367] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240502 08:18:07.548337 16143 raft_consensus.cc:348] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.548756 16143 raft_consensus.cc:374] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.548920 16143 raft_consensus.cc:729] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.549330 16143 consensus_queue.cc:260] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.550179 16143 ts_tablet_manager.cc:1398] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:07.551542 16141 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: No bootstrap required, opened a new log
I20240502 08:18:07.551717 16141 ts_tablet_manager.cc:1367] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 08:18:07.552274 16141 raft_consensus.cc:348] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.552451 16141 raft_consensus.cc:374] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.552529 16141 raft_consensus.cc:729] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.552776 16141 consensus_queue.cc:260] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.553900 16141 ts_tablet_manager.cc:1398] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:07.630311 16147 raft_consensus.cc:482] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.630504 16147 raft_consensus.cc:504] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.631098 16147 leader_election.cc:288] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:07.631726 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:07.631974 15235 raft_consensus.cc:2454] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 0.
I20240502 08:18:07.632432 15041 leader_election.cc:302] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:07.632714 16147 raft_consensus.cc:2781] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.632822 16147 raft_consensus.cc:482] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.632922 16147 raft_consensus.cc:3037] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.634790 16147 raft_consensus.cc:504] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.635212 16147 leader_election.cc:288] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:07.635850 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:07.636075 15235 raft_consensus.cc:3037] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.637904 15235 raft_consensus.cc:2454] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 1.
I20240502 08:18:07.638330 15041 leader_election.cc:302] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:07.638603 16147 raft_consensus.cc:2781] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.638991 16147 raft_consensus.cc:686] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:18:07.639288 16147 consensus_queue.cc:237] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } }
I20240502 08:18:07.641659 14833 catalog_manager.cc:5484] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } }
I20240502 08:18:07.714872 16129 raft_consensus.cc:482] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.715051 16129 raft_consensus.cc:504] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.715680 16129 leader_election.cc:288] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:07.716210 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445454fd0e3d405d87fc7ef20929e862" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29" is_pre_election: true
I20240502 08:18:07.716430 15235 raft_consensus.cc:2454] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 0.
I20240502 08:18:07.716797 14910 leader_election.cc:302] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:07.717077 16129 raft_consensus.cc:2781] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.717183 16129 raft_consensus.cc:482] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.717274 16129 raft_consensus.cc:3037] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.719197 16129 raft_consensus.cc:504] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.719645 16129 leader_election.cc:288] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:07.720163 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445454fd0e3d405d87fc7ef20929e862" candidate_uuid: "97037d47bf1249d2b5932b6553b907d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:07.720346 15235 raft_consensus.cc:3037] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.722199 15235 raft_consensus.cc:2454] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97037d47bf1249d2b5932b6553b907d4 in term 1.
I20240502 08:18:07.722635 14910 leader_election.cc:302] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:07.722985 16129 raft_consensus.cc:2781] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.723405 16129 raft_consensus.cc:686] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Leader. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:07.723701 16129 consensus_queue.cc:237] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } }
I20240502 08:18:07.726330 14833 catalog_manager.cc:5484] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } }
I20240502 08:18:07.792395 16147 raft_consensus.cc:482] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.792640 16147 raft_consensus.cc:504] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.793300 16147 leader_election.cc:288] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:07.799312 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4" is_pre_election: true
I20240502 08:18:07.799561 14975 raft_consensus.cc:2454] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 0.
I20240502 08:18:07.800050 15042 leader_election.cc:302] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:07.800359 16147 raft_consensus.cc:2781] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.800475 16147 raft_consensus.cc:482] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.800591 16147 raft_consensus.cc:3037] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.802762 16147 raft_consensus.cc:504] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.803233 16147 leader_election.cc:288] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:07.803722 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e" candidate_uuid: "7fe0e31240d145a28e75262f706ebb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:18:07.803908 14975 raft_consensus.cc:3037] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.805730 14975 raft_consensus.cc:2454] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe0e31240d145a28e75262f706ebb89 in term 1.
I20240502 08:18:07.806123 15042 leader_election.cc:302] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 97037d47bf1249d2b5932b6553b907d4; no voters:
I20240502 08:18:07.806412 16147 raft_consensus.cc:2781] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.806568 16147 raft_consensus.cc:686] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:18:07.806785 16147 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } }
I20240502 08:18:07.809219 14833 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } }
W20240502 08:18:08.155767 16158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:08.156092 16158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:08.163615 16158 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:18:08.251000 16147 consensus_queue.cc:1035] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.254771 16129 consensus_queue.cc:1035] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.340557 14975 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.341277 16151 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.360392 16192 mvcc.cc:205] Tried to move back new op lower bound from 7023156790634885120 to 7023156788457963520. Current Snapshot: MvccSnapshot[applied={T|T < 7023156790634885120}]
INSERT report
rows total: 2000
time total: 182.726 ms
time per row: 0.0913629 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:18:09.433777 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:44799
--local_ip_for_outbound_sockets=127.8.68.196
--tserver_master_addrs=127.8.68.254:44229
--webserver_port=39553
--webserver_interface=127.8.68.196
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:18:09.782334 16205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:09.782650 16205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:09.782795 16205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:09.790432 16205 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:09.790599 16205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:09.790764 16205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240502 08:18:09.800021 16205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:44799
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=39553
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:09.801440 16205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:09.803411 16205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:09.816655 16211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.816655 16213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:09.817329 16210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:09.817852 16205 server_base.cc:1034] running on GCE node
I20240502 08:18:09.818481 16205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:09.819567 16205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:09.820781 16205 hybrid_clock.cc:648] HybridClock initialized: now 1714637889820718 us; error 51 us; skew 500 ppm
I20240502 08:18:09.821127 16205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:09.828110 16205 webserver.cc:460] Webserver started at http://127.8.68.196:39553/ using document root <none> and password file <none>
I20240502 08:18:09.828882 16205 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:09.829030 16205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:09.834563 16205 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:18:09.837545 16219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:09.839031 16205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:09.839203 16205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:18:09.840162 16205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:09.859683 16205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:09.860515 16205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:09.860801 16205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:09.861699 16205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:09.863626 16226 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 08:18:09.869678 16205 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 08:18:09.869802 16205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:09.869946 16205 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 08:18:09.873195 16205 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 08:18:09.873303 16205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:09.873701 16226 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:18:09.893218 16226 log.cc:824] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:09.901911 16226 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:09.902555 16226 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap complete.
I20240502 08:18:09.904001 16226 ts_tablet_manager.cc:1367] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.016s
I20240502 08:18:09.913938 16226 raft_consensus.cc:348] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:09.914769 16226 raft_consensus.cc:729] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.915638 16226 consensus_queue.cc:260] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:09.917492 16226 ts_tablet_manager.cc:1398] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240502 08:18:09.917989 16205 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:44799
I20240502 08:18:09.918051 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:44799 every 8 connection(s)
I20240502 08:18:09.919363 16205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
W20240502 08:18:09.920559 16336 tablet.cc:2369] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:09.923820 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 16205
I20240502 08:18:09.924492 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:41747
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:44229
--webserver_port=46347
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:18:09.932662 16335 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:18:09.932979 16335 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:09.933717 16335 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:18:09.936146 14828 ts_manager.cc:194] Re-registered known tserver with Master: 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:18:09.937386 14828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:43651
W20240502 08:18:10.289873 16339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:10.290206 16339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:10.290349 16339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:10.298025 16339 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:10.298197 16339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:10.298382 16339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:18:10.307968 16339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:41747
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=46347
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:10.309343 16339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:10.311110 16339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:10.323115 16344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.323334 16345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.323735 16347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:10.323978 16339 server_base.cc:1034] running on GCE node
I20240502 08:18:10.324607 16339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:10.325544 16339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:10.326761 16339 hybrid_clock.cc:648] HybridClock initialized: now 1714637890326699 us; error 58 us; skew 500 ppm
I20240502 08:18:10.327069 16339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:10.329439 16339 webserver.cc:460] Webserver started at http://127.8.68.197:46347/ using document root <none> and password file <none>
I20240502 08:18:10.330191 16339 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:10.330304 16339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:10.335951 16339 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 08:18:10.338793 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.340384 16339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:10.340548 16339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
format_stamp: "Formatted at 2024-05-02 08:17:55 on dist-test-slave-2dhs"
I20240502 08:18:10.341392 16339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:10.360090 16339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:10.360936 16339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:10.361203 16339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:10.362134 16339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:10.363804 16339 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:10.363886 16339 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.363974 16339 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:10.364017 16339 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.397212 16339 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:41747
I20240502 08:18:10.397241 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:41747 every 8 connection(s)
I20240502 08:18:10.398897 16339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:18:10.408468 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 16339
I20240502 08:18:10.409098 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:34645
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:44229
--webserver_port=40219
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:18:10.412556 16466 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:18:10.412876 16466 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:10.413640 16466 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:18:10.415024 14828 ts_manager.cc:194] Re-registered known tserver with Master: 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747)
I20240502 08:18:10.415855 14828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:53847
W20240502 08:18:10.748557 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:10.748948 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:10.749081 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:10.761477 16469 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:10.761619 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:10.761734 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:18:10.771008 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:34645
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=40219
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:10.772500 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:10.774469 16469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:10.787870 16474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.788122 16475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:10.788225 16477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:10.788892 16469 server_base.cc:1034] running on GCE node
I20240502 08:18:10.789546 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:10.790592 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:10.791806 16469 hybrid_clock.cc:648] HybridClock initialized: now 1714637890791768 us; error 53 us; skew 500 ppm
I20240502 08:18:10.792160 16469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:10.794735 16469 webserver.cc:460] Webserver started at http://127.8.68.198:40219/ using document root <none> and password file <none>
I20240502 08:18:10.795667 16469 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:10.795826 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:10.801867 16469 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 08:18:10.805296 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:10.806970 16469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:10.807179 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a8cf9c3ffc34440594482a6718b596e5"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:18:10.808202 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:10.833527 16469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:10.834414 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:10.834710 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:10.835788 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:10.837848 16490 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 08:18:10.843542 16469 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 08:18:10.843693 16469 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:10.843849 16469 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 08:18:10.847088 16469 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 08:18:10.847218 16469 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240502 08:18:10.847646 16490 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Bootstrap starting.
I20240502 08:18:10.868081 16490 log.cc:824] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:10.871908 16490 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:10.872730 16490 tablet_bootstrap.cc:492] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Bootstrap complete.
I20240502 08:18:10.874037 16490 ts_tablet_manager.cc:1367] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.012s
I20240502 08:18:10.884125 16490 raft_consensus.cc:348] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:10.885668 16490 raft_consensus.cc:729] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.886574 16490 consensus_queue.cc:260] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:10.888540 16490 ts_tablet_manager.cc:1398] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20240502 08:18:10.897294 16469 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:34645
I20240502 08:18:10.897378 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:34645 every 8 connection(s)
I20240502 08:18:10.898833 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:18:10.900651 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 16469
I20240502 08:18:10.901238 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:34535
--local_ip_for_outbound_sockets=127.8.68.199
--tserver_master_addrs=127.8.68.254:44229
--webserver_port=44259
--webserver_interface=127.8.68.199
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:18:10.900256 16600 tablet.cc:2369] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:10.912336 16599 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:18:10.912657 16599 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:10.913528 16599 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:18:10.916096 14828 ts_manager.cc:194] Re-registered known tserver with Master: a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645)
I20240502 08:18:10.918114 14828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:35791
I20240502 08:18:10.940352 16335 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
W20240502 08:18:11.255151 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:11.255510 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:11.255614 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:11.263268 16602 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:11.263427 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:11.263550 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:18:11.272972 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:34535
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=44259
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:11.274487 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:11.276597 16602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:11.289491 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:11.289733 16610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:11.289952 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:11.290284 16602 server_base.cc:1034] running on GCE node
I20240502 08:18:11.290933 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:11.291962 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:11.293177 16602 hybrid_clock.cc:648] HybridClock initialized: now 1714637891293128 us; error 46 us; skew 500 ppm
I20240502 08:18:11.293522 16602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:11.295894 16602 webserver.cc:460] Webserver started at http://127.8.68.199:44259/ using document root <none> and password file <none>
I20240502 08:18:11.296679 16602 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:11.296815 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:11.302542 16602 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 08:18:11.305822 16616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:11.307204 16602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:11.307456 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7757b11bdda3443199e83298ba5c8118"
format_stamp: "Formatted at 2024-05-02 08:17:56 on dist-test-slave-2dhs"
I20240502 08:18:11.308300 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:11.336234 16602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:11.337081 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:11.337368 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:11.338369 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:11.340044 16602 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:11.340132 16602 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:11.340277 16602 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:11.340330 16602 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:11.373673 16602 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:34535
I20240502 08:18:11.373720 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:34535 every 8 connection(s)
I20240502 08:18:11.375175 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:18:11.383708 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 16602
I20240502 08:18:11.384039 8467 test_util.cc:268] Using random seed: 1047474878
I20240502 08:18:11.389179 16729 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:18:11.389532 16729 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:11.390396 16729 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:18:11.392025 14828 ts_manager.cc:194] Re-registered known tserver with Master: 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535)
I20240502 08:18:11.392789 14828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:42583
I20240502 08:18:11.401862 8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 08:18:11.402333 8467 test_util.cc:268] Using random seed: 1047493173
I20240502 08:18:11.418385 16466 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
I20240502 08:18:11.418969 8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 08:18:11.419476 8467 test_util.cc:268] Using random seed: 1047510294
I20240502 08:18:11.445246 16759 raft_consensus.cc:482] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.445591 16759 raft_consensus.cc:504] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:11.447613 16759 leader_election.cc:288] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645)
I20240502 08:18:11.450039 8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 08:18:11.460484 8467 test_util.cc:268] Using random seed: 1047551301
I20240502 08:18:11.471578 16553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed128f6b7a67452db6b11ef0d14937dc" candidate_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8cf9c3ffc34440594482a6718b596e5" is_pre_election: true
I20240502 08:18:11.472178 16553 raft_consensus.cc:2396] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83659c8dbbcd4142abdbcb243eeec8d4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:18:11.473985 16221 leader_election.cc:302] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 83659c8dbbcd4142abdbcb243eeec8d4; no voters: a8cf9c3ffc34440594482a6718b596e5
I20240502 08:18:11.475062 16759 raft_consensus.cc:2726] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:11.528198 8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 08:18:11.540356 8467 test_util.cc:268] Using random seed: 1047631160
I20240502 08:18:11.602725 8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 08:18:11.922765 16599 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
I20240502 08:18:12.395887 16729 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
I20240502 08:18:12.578514 16848 raft_consensus.cc:482] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.578991 16848 raft_consensus.cc:504] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:12.591789 16848 leader_election.cc:288] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:18:12.683096 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed128f6b7a67452db6b11ef0d14937dc" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" is_pre_election: true
I20240502 08:18:12.683535 16289 raft_consensus.cc:2454] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 1.
I20240502 08:18:12.685060 16484 leader_election.cc:302] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 83659c8dbbcd4142abdbcb243eeec8d4, a8cf9c3ffc34440594482a6718b596e5; no voters:
I20240502 08:18:12.685587 16848 raft_consensus.cc:2781] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:12.685755 16848 raft_consensus.cc:482] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.685859 16848 raft_consensus.cc:3037] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.691022 16848 raft_consensus.cc:504] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:12.693362 16848 leader_election.cc:288] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 election: Requested vote from peers 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:18:12.694356 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed128f6b7a67452db6b11ef0d14937dc" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
I20240502 08:18:12.694588 16289 raft_consensus.cc:3037] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:12.700337 16289 raft_consensus.cc:2454] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 2.
I20240502 08:18:12.700939 16484 leader_election.cc:302] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 83659c8dbbcd4142abdbcb243eeec8d4, a8cf9c3ffc34440594482a6718b596e5; no voters:
I20240502 08:18:12.701500 16848 raft_consensus.cc:2781] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:12.767359 16848 raft_consensus.cc:686] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [term 2 LEADER]: Becoming Leader. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Running, Role: LEADER
I20240502 08:18:12.768062 16848 consensus_queue.cc:237] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:12.776672 14829 catalog_manager.cc:5484] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a8cf9c3ffc34440594482a6718b596e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:13.209720 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:13.210336 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:13.220410 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:13.300545 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:13.333009 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:13.354919 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:13.406141 16289 raft_consensus.cc:1261] T ed128f6b7a67452db6b11ef0d14937dc P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Refusing update from remote peer a8cf9c3ffc34440594482a6718b596e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:18:13.407125 16894 consensus_queue.cc:1035] T ed128f6b7a67452db6b11ef0d14937dc P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.414526 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:14.246954 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:14.276954 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:14.288514 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:14.324388 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:14.344002 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:14.411393 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:14.465631 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:15.351068 14974 consensus_queue.cc:237] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:15.360132 15106 raft_consensus.cc:1261] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 08:18:15.361124 16947 consensus_queue.cc:1035] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 08:18:15.365092 16804 raft_consensus.cc:2932] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:15.374248 15106 raft_consensus.cc:2932] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:15.375993 14829 catalog_manager.cc:5484] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 66, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
W20240502 08:18:15.456979 14908 consensus_peers.cc:485] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747): Couldn't send request to peer 382fd0c6daaf4eddb1c57557e9b716d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c94e28f8f2b47cba77ed459c8e731d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.696807 15233 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:15.724233 14975 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240502 08:18:15.725278 16956 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240502 08:18:15.730300 16767 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:15.736662 14828 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index -1 to 67, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
W20240502 08:18:15.749284 15172 consensus_peers.cc:485] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535): Couldn't send request to peer 7757b11bdda3443199e83298ba5c8118. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2f290f16fe044658ed9b6b9a0a71f08. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.751739 14975 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:15.893508 14975 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:15.897145 15236 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 08:18:15.914678 16757 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 08:18:15.918777 16757 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
W20240502 08:18:15.926429 14909 consensus_peers.cc:485] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 -> Peer a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645): Couldn't send request to peer a8cf9c3ffc34440594482a6718b596e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4485a1686beb45d7ad56010d2a7c9fc9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.929030 15233 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:15.930860 14828 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 72, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:16.010277 17028 ts_tablet_manager.cc:902] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:16.116926 17035 ts_tablet_manager.cc:902] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Initiating tablet copy from peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:16.198522 17028 tablet_copy_client.cc:308] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:16.309136 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 4c94e28f8f2b47cba77ed459c8e731d0 from peer 382fd0c6daaf4eddb1c57557e9b716d1 ({username='slave'} at 127.8.68.197:56427)
I20240502 08:18:16.309502 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 4c94e28f8f2b47cba77ed459c8e731d0 from peer 382fd0c6daaf4eddb1c57557e9b716d1 at {username='slave'} at 127.8.68.197:56427: session id = 382fd0c6daaf4eddb1c57557e9b716d1-4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:18:16.321522 14995 tablet_copy_source_session.cc:205] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.327481 17035 tablet_copy_client.cc:308] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:43799
I20240502 08:18:16.329129 17028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c94e28f8f2b47cba77ed459c8e731d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.389569 15256 tablet_copy_service.cc:140] P 19d550ab72684bf8aa1382d9c348cd29: Received BeginTabletCopySession request for tablet b2f290f16fe044658ed9b6b9a0a71f08 from peer 7757b11bdda3443199e83298ba5c8118 ({username='slave'} at 127.8.68.199:33267)
I20240502 08:18:16.389883 15256 tablet_copy_service.cc:161] P 19d550ab72684bf8aa1382d9c348cd29: Beginning new tablet copy session on tablet b2f290f16fe044658ed9b6b9a0a71f08 from peer 7757b11bdda3443199e83298ba5c8118 at {username='slave'} at 127.8.68.199:33267: session id = 7757b11bdda3443199e83298ba5c8118-b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:18:16.394496 15256 tablet_copy_source_session.cc:205] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.410282 17035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f290f16fe044658ed9b6b9a0a71f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.422741 17028 tablet_copy_client.cc:783] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.423238 17028 tablet_copy_client.cc:647] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.441699 17028 tablet_copy_client.cc:523] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.458917 17028 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap starting.
I20240502 08:18:16.474421 17035 tablet_copy_client.cc:783] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.474850 17035 tablet_copy_client.cc:647] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.477793 17035 tablet_copy_client.cc:523] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.482668 17035 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Bootstrap starting.
I20240502 08:18:16.493781 17053 ts_tablet_manager.cc:902] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:16.534459 17028 log.cc:824] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:16.538396 17035 log.cc:824] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:16.542495 17053 tablet_copy_client.cc:308] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:16.549170 15102 consensus_queue.cc:237] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:16.556803 15236 raft_consensus.cc:1261] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 08:18:16.557670 16836 consensus_queue.cc:1035] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 08:18:16.561178 16836 raft_consensus.cc:2932] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index -1 to 93, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:16.564116 15233 raft_consensus.cc:2932] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index -1 to 93, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:16.564716 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 4485a1686beb45d7ad56010d2a7c9fc9 from peer a8cf9c3ffc34440594482a6718b596e5 ({username='slave'} at 127.8.68.198:60973)
I20240502 08:18:16.564905 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 4485a1686beb45d7ad56010d2a7c9fc9 from peer a8cf9c3ffc34440594482a6718b596e5 at {username='slave'} at 127.8.68.198:60973: session id = a8cf9c3ffc34440594482a6718b596e5-4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:18:16.567745 14995 tablet_copy_source_session.cc:205] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.570524 17053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4485a1686beb45d7ad56010d2a7c9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.577250 14828 catalog_manager.cc:5484] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index -1 to 93, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:16.579525 17053 tablet_copy_client.cc:783] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.579879 17053 tablet_copy_client.cc:647] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.582854 17053 tablet_copy_client.cc:523] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.586722 17053 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap starting.
W20240502 08:18:16.702749 15039 consensus_peers.cc:485] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 -> Peer 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799): Couldn't send request to peer 83659c8dbbcd4142abdbcb243eeec8d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aa2b2b96e134484a50edaac02dcc2a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:16.882211 17028 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.882890 17028 tablet_bootstrap.cc:492] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap complete.
I20240502 08:18:16.891191 17028 ts_tablet_manager.cc:1367] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent bootstrapping tablet: real 0.435s user 0.140s sys 0.030s
I20240502 08:18:17.131109 17053 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=720 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:17.132594 17053 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap complete.
I20240502 08:18:17.133206 17028 raft_consensus.cc:348] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:17.133736 17028 raft_consensus.cc:729] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382fd0c6daaf4eddb1c57557e9b716d1, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.135494 17053 ts_tablet_manager.cc:1367] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Time spent bootstrapping tablet: real 0.549s user 0.177s sys 0.052s
I20240502 08:18:17.137015 17053 raft_consensus.cc:348] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:17.138729 17053 raft_consensus.cc:729] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.139403 17053 consensus_queue.cc:260] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:17.140317 17053 ts_tablet_manager.cc:1398] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:17.141767 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session a8cf9c3ffc34440594482a6718b596e5-4485a1686beb45d7ad56010d2a7c9fc9 received from {username='slave'} at 127.8.68.198:60973
I20240502 08:18:17.142009 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session a8cf9c3ffc34440594482a6718b596e5-4485a1686beb45d7ad56010d2a7c9fc9 on tablet 4485a1686beb45d7ad56010d2a7c9fc9 with peer a8cf9c3ffc34440594482a6718b596e5
I20240502 08:18:17.149915 17028 consensus_queue.cc:260] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:17.185405 17091 ts_tablet_manager.cc:902] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Initiating tablet copy from peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:17.212697 17028 ts_tablet_manager.cc:1398] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent starting tablet: real 0.321s user 0.015s sys 0.008s
I20240502 08:18:17.215215 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-4c94e28f8f2b47cba77ed459c8e731d0 received from {username='slave'} at 127.8.68.197:56427
I20240502 08:18:17.215382 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-4c94e28f8f2b47cba77ed459c8e731d0 on tablet 4c94e28f8f2b47cba77ed459c8e731d0 with peer 382fd0c6daaf4eddb1c57557e9b716d1
I20240502 08:18:17.354774 17035 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=79 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:17.355567 17035 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Bootstrap complete.
I20240502 08:18:17.356118 17035 ts_tablet_manager.cc:1367] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Time spent bootstrapping tablet: real 0.874s user 0.150s sys 0.028s
I20240502 08:18:17.357532 17091 tablet_copy_client.cc:308] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:44789
I20240502 08:18:17.363032 17035 raft_consensus.cc:348] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:17.390655 17035 raft_consensus.cc:729] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7757b11bdda3443199e83298ba5c8118, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.391721 17035 consensus_queue.cc:260] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:17.441967 15104 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:17.446002 17035 ts_tablet_manager.cc:1398] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118: Time spent starting tablet: real 0.090s user 0.009s sys 0.028s
I20240502 08:18:17.447723 15256 tablet_copy_service.cc:342] P 19d550ab72684bf8aa1382d9c348cd29: Request end of tablet copy session 7757b11bdda3443199e83298ba5c8118-b2f290f16fe044658ed9b6b9a0a71f08 received from {username='slave'} at 127.8.68.199:33267
I20240502 08:18:17.447901 15256 tablet_copy_service.cc:433] P 19d550ab72684bf8aa1382d9c348cd29: ending tablet copy session 7757b11bdda3443199e83298ba5c8118-b2f290f16fe044658ed9b6b9a0a71f08 on tablet b2f290f16fe044658ed9b6b9a0a71f08 with peer 7757b11bdda3443199e83298ba5c8118
I20240502 08:18:17.453363 14972 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 08:18:17.456398 17080 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 08:18:17.457368 15126 tablet_copy_service.cc:140] P 7fe0e31240d145a28e75262f706ebb89: Received BeginTabletCopySession request for tablet 1aa2b2b96e134484a50edaac02dcc2a3 from peer 83659c8dbbcd4142abdbcb243eeec8d4 ({username='slave'} at 127.8.68.196:33365)
I20240502 08:18:17.457648 15126 tablet_copy_service.cc:161] P 7fe0e31240d145a28e75262f706ebb89: Beginning new tablet copy session on tablet 1aa2b2b96e134484a50edaac02dcc2a3 from peer 83659c8dbbcd4142abdbcb243eeec8d4 at {username='slave'} at 127.8.68.196:33365: session id = 83659c8dbbcd4142abdbcb243eeec8d4-1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:18:17.466598 15126 tablet_copy_source_session.cc:205] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:17.472285 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa2b2b96e134484a50edaac02dcc2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.477428 14972 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:17.477411 17080 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:17.484763 14829 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 129, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:17.495728 17091 tablet_copy_client.cc:783] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:17.496094 17091 tablet_copy_client.cc:647] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 1 WAL segments...
W20240502 08:18:17.502796 15039 consensus_peers.cc:485] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 -> Peer 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747): Couldn't send request to peer 382fd0c6daaf4eddb1c57557e9b716d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e2e9479648049ca94e7e5bfeb98db6e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:17.511222 17091 tablet_copy_client.cc:523] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:17.514945 17091 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:18:17.557956 16684 raft_consensus.cc:1203] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.96] Dedup: 1.80->[1.81-1.96]
I20240502 08:18:17.569967 16553 raft_consensus.cc:1203] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.108] Dedup: 1.89->[1.90-1.108]
I20240502 08:18:17.592407 16421 raft_consensus.cc:1203] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.102] Dedup: 1.78->[1.79-1.102]
I20240502 08:18:18.116761 17140 ts_tablet_manager.cc:902] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Initiating tablet copy from peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:18.157850 17140 tablet_copy_client.cc:308] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:44789
I20240502 08:18:18.202222 15126 tablet_copy_service.cc:140] P 7fe0e31240d145a28e75262f706ebb89: Received BeginTabletCopySession request for tablet 6e2e9479648049ca94e7e5bfeb98db6e from peer 382fd0c6daaf4eddb1c57557e9b716d1 ({username='slave'} at 127.8.68.197:47443)
I20240502 08:18:18.202454 15126 tablet_copy_service.cc:161] P 7fe0e31240d145a28e75262f706ebb89: Beginning new tablet copy session on tablet 6e2e9479648049ca94e7e5bfeb98db6e from peer 382fd0c6daaf4eddb1c57557e9b716d1 at {username='slave'} at 127.8.68.197:47443: session id = 382fd0c6daaf4eddb1c57557e9b716d1-6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:18:18.206485 15126 tablet_copy_source_session.cc:205] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:18.208621 17140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2e9479648049ca94e7e5bfeb98db6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.216264 15235 consensus_queue.cc:237] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:18.225149 17140 tablet_copy_client.cc:783] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:18.225471 17140 tablet_copy_client.cc:647] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:18.231451 17140 tablet_copy_client.cc:523] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:18:18.244765 15172 consensus_peers.cc:485] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535): Couldn't send request to peer 7757b11bdda3443199e83298ba5c8118. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e7ca0fbedbe43e2a42fd849c233ded3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.249945 17140 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap starting.
I20240502 08:18:18.276175 15103 raft_consensus.cc:1261] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240502 08:18:18.277040 16956 consensus_queue.cc:1035] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 08:18:18.301412 16782 raft_consensus.cc:2932] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:18.309212 15104 raft_consensus.cc:2932] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:18.314016 14828 catalog_manager.cc:5484] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index -1 to 123, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:18.345057 17091 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:18.348599 17091 tablet_bootstrap.cc:492] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap complete.
I20240502 08:18:18.350910 17091 ts_tablet_manager.cc:1367] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 0.836s user 0.179s sys 0.019s
I20240502 08:18:18.353546 17091 raft_consensus.cc:348] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:18.354789 17091 raft_consensus.cc:729] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.355394 17091 consensus_queue.cc:260] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:18.367280 17091 ts_tablet_manager.cc:1398] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20240502 08:18:18.371953 15126 tablet_copy_service.cc:342] P 7fe0e31240d145a28e75262f706ebb89: Request end of tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-1aa2b2b96e134484a50edaac02dcc2a3 received from {username='slave'} at 127.8.68.196:33365
I20240502 08:18:18.372146 15126 tablet_copy_service.cc:433] P 7fe0e31240d145a28e75262f706ebb89: ending tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-1aa2b2b96e134484a50edaac02dcc2a3 on tablet 1aa2b2b96e134484a50edaac02dcc2a3 with peer 83659c8dbbcd4142abdbcb243eeec8d4
I20240502 08:18:18.621881 16289 raft_consensus.cc:1203] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.92->[1.93-1.112] Dedup: 1.100->[1.101-1.112]
I20240502 08:18:18.643971 14974 consensus_queue.cc:237] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:18.650833 17180 ts_tablet_manager.cc:902] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Initiating tablet copy from peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
W20240502 08:18:18.665112 14909 consensus_peers.cc:485] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 -> Peer a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645): Couldn't send request to peer a8cf9c3ffc34440594482a6718b596e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 445454fd0e3d405d87fc7ef20929e862. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.668584 17180 tablet_copy_client.cc:308] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:43799
I20240502 08:18:18.669723 15256 tablet_copy_service.cc:140] P 19d550ab72684bf8aa1382d9c348cd29: Received BeginTabletCopySession request for tablet 4e7ca0fbedbe43e2a42fd849c233ded3 from peer 7757b11bdda3443199e83298ba5c8118 ({username='slave'} at 127.8.68.199:33267)
I20240502 08:18:18.669935 15256 tablet_copy_service.cc:161] P 19d550ab72684bf8aa1382d9c348cd29: Beginning new tablet copy session on tablet 4e7ca0fbedbe43e2a42fd849c233ded3 from peer 7757b11bdda3443199e83298ba5c8118 at {username='slave'} at 127.8.68.199:33267: session id = 7757b11bdda3443199e83298ba5c8118-4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:18:18.673557 15256 tablet_copy_source_session.cc:205] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:18.675557 17180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca0fbedbe43e2a42fd849c233ded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.685621 15234 raft_consensus.cc:1261] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 08:18:18.695236 17180 tablet_copy_client.cc:783] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:18.695647 17180 tablet_copy_client.cc:647] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:18.700318 16804 consensus_queue.cc:1035] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 08:18:18.710613 17180 tablet_copy_client.cc:523] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:18.710496 16129 raft_consensus.cc:2932] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index -1 to 114, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:18.711189 15233 raft_consensus.cc:2932] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:18.715916 17180 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Bootstrap starting.
I20240502 08:18:18.720443 14829 catalog_manager.cc:5484] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 114, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:18.783116 17140 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=143 ignored=0} inserts{seen=828 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:18.783726 17140 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap complete.
I20240502 08:18:18.784137 17140 ts_tablet_manager.cc:1367] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent bootstrapping tablet: real 0.534s user 0.217s sys 0.014s
I20240502 08:18:18.785264 17140 raft_consensus.cc:348] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:18.785589 17140 raft_consensus.cc:729] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382fd0c6daaf4eddb1c57557e9b716d1, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.785928 17140 consensus_queue.cc:260] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 143, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:19.137028 17140 ts_tablet_manager.cc:1398] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent starting tablet: real 0.353s user 0.053s sys 0.088s
I20240502 08:18:19.142792 15126 tablet_copy_service.cc:342] P 7fe0e31240d145a28e75262f706ebb89: Request end of tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-6e2e9479648049ca94e7e5bfeb98db6e received from {username='slave'} at 127.8.68.197:47443
I20240502 08:18:19.142939 15126 tablet_copy_service.cc:433] P 7fe0e31240d145a28e75262f706ebb89: ending tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-6e2e9479648049ca94e7e5bfeb98db6e on tablet 6e2e9479648049ca94e7e5bfeb98db6e with peer 382fd0c6daaf4eddb1c57557e9b716d1
I20240502 08:18:19.181002 17203 ts_tablet_manager.cc:902] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:19.316679 17203 tablet_copy_client.cc:308] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:19.317535 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 445454fd0e3d405d87fc7ef20929e862 from peer a8cf9c3ffc34440594482a6718b596e5 ({username='slave'} at 127.8.68.198:60973)
I20240502 08:18:19.317708 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 445454fd0e3d405d87fc7ef20929e862 from peer a8cf9c3ffc34440594482a6718b596e5 at {username='slave'} at 127.8.68.198:60973: session id = a8cf9c3ffc34440594482a6718b596e5-445454fd0e3d405d87fc7ef20929e862
I20240502 08:18:19.318955 14972 consensus_queue.cc:237] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:19.320973 14995 tablet_copy_source_session.cc:205] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.323612 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445454fd0e3d405d87fc7ef20929e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.325309 15103 raft_consensus.cc:1261] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240502 08:18:19.328330 16129 consensus_queue.cc:1035] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240502 08:18:19.331710 16129 raft_consensus.cc:2932] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index -1 to 139, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:19.336313 15103 raft_consensus.cc:2932] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index -1 to 139, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:19.345080 14828 catalog_manager.cc:5484] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 139, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:19.367275 17203 tablet_copy_client.cc:783] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.367614 17203 tablet_copy_client.cc:647] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.384498 17203 tablet_copy_client.cc:523] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.395864 17203 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap starting.
W20240502 08:18:19.476333 14908 consensus_peers.cc:485] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 -> Peer 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799): Couldn't send request to peer 83659c8dbbcd4142abdbcb243eeec8d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68b0f3cfa17644449993f63affcbe2be. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.585229 16421 raft_consensus.cc:1203] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.128->[1.129-1.166] Dedup: 1.143->[1.144-1.166]
I20240502 08:18:19.882611 15232 consensus_queue.cc:237] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.150, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:19.894131 15102 raft_consensus.cc:1261] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240502 08:18:19.897531 17205 consensus_queue.cc:1035] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240502 08:18:19.915710 17245 ts_tablet_manager.cc:902] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
W20240502 08:18:19.922300 15169 consensus_peers.cc:485] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747): Couldn't send request to peer 382fd0c6daaf4eddb1c57557e9b716d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df91c5f0a46c43d78f8f7aeef7281e14. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.956077 17180 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Bootstrap replayed 1/1 log segments. Stats: ops{read=126 overwritten=0 applied=126 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:19.956637 17180 tablet_bootstrap.cc:492] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Bootstrap complete.
I20240502 08:18:19.956969 17180 ts_tablet_manager.cc:1367] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Time spent bootstrapping tablet: real 1.241s user 0.190s sys 0.011s
I20240502 08:18:19.957940 17180 raft_consensus.cc:348] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:19.958259 17180 raft_consensus.cc:729] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7757b11bdda3443199e83298ba5c8118, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.958611 17180 consensus_queue.cc:260] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 126, Last appended: 1.126, Last appended by leader: 126, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:19.962194 17205 raft_consensus.cc:2932] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.151: config changed from index -1 to 151, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:19.964022 17245 tablet_copy_client.cc:308] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:19.965942 17180 ts_tablet_manager.cc:1398] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 08:18:19.969089 15256 tablet_copy_service.cc:342] P 19d550ab72684bf8aa1382d9c348cd29: Request end of tablet copy session 7757b11bdda3443199e83298ba5c8118-4e7ca0fbedbe43e2a42fd849c233ded3 received from {username='slave'} at 127.8.68.199:33267
I20240502 08:18:19.969218 15256 tablet_copy_service.cc:433] P 19d550ab72684bf8aa1382d9c348cd29: ending tablet copy session 7757b11bdda3443199e83298ba5c8118-4e7ca0fbedbe43e2a42fd849c233ded3 on tablet 4e7ca0fbedbe43e2a42fd849c233ded3 with peer 7757b11bdda3443199e83298ba5c8118
I20240502 08:18:19.970814 14829 catalog_manager.cc:5484] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index -1 to 151, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:19.981014 15104 raft_consensus.cc:2932] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index -1 to 151, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:20.012951 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 68b0f3cfa17644449993f63affcbe2be from peer 83659c8dbbcd4142abdbcb243eeec8d4 ({username='slave'} at 127.8.68.196:60007)
I20240502 08:18:20.013155 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 68b0f3cfa17644449993f63affcbe2be from peer 83659c8dbbcd4142abdbcb243eeec8d4 at {username='slave'} at 127.8.68.196:60007: session id = 83659c8dbbcd4142abdbcb243eeec8d4-68b0f3cfa17644449993f63affcbe2be
I20240502 08:18:20.015987 14995 tablet_copy_source_session.cc:205] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.017663 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b0f3cfa17644449993f63affcbe2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.024114 17245 tablet_copy_client.cc:783] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.024888 17245 tablet_copy_client.cc:647] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.027443 17245 tablet_copy_client.cc:523] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.040912 17245 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:18:20.153204 17203 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=122 overwritten=0 applied=122 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.153915 17203 tablet_bootstrap.cc:492] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Bootstrap complete.
I20240502 08:18:20.154295 17203 ts_tablet_manager.cc:1367] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Time spent bootstrapping tablet: real 0.759s user 0.175s sys 0.022s
I20240502 08:18:20.155717 17203 raft_consensus.cc:348] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:20.156211 17203 raft_consensus.cc:729] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.156527 17203 consensus_queue.cc:260] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:20.158025 17203 ts_tablet_manager.cc:1398] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:20.167799 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session a8cf9c3ffc34440594482a6718b596e5-445454fd0e3d405d87fc7ef20929e862 received from {username='slave'} at 127.8.68.198:60973
I20240502 08:18:20.167944 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session a8cf9c3ffc34440594482a6718b596e5-445454fd0e3d405d87fc7ef20929e862 on tablet 445454fd0e3d405d87fc7ef20929e862 with peer a8cf9c3ffc34440594482a6718b596e5
I20240502 08:18:20.169080 16552 raft_consensus.cc:1203] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.113->[1.114-1.136] Dedup: 1.122->[1.123-1.136]
I20240502 08:18:20.364514 17268 ts_tablet_manager.cc:902] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Initiating tablet copy from peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:20.410952 16684 raft_consensus.cc:1203] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.122->[1.123-1.158] Dedup: 1.126->[1.127-1.158]
I20240502 08:18:20.465736 17245 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=149 overwritten=0 applied=149 ignored=0} inserts{seen=834 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.466344 17245 tablet_bootstrap.cc:492] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap complete.
I20240502 08:18:20.466717 17245 ts_tablet_manager.cc:1367] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 0.426s user 0.205s sys 0.038s
I20240502 08:18:20.467653 17245 raft_consensus.cc:348] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:20.467967 17245 raft_consensus.cc:729] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.468292 17245 consensus_queue.cc:260] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
W20240502 08:18:20.473810 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 206493us
I20240502 08:18:20.476064 17245 ts_tablet_manager.cc:1398] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240502 08:18:20.479928 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-68b0f3cfa17644449993f63affcbe2be received from {username='slave'} at 127.8.68.196:60007
I20240502 08:18:20.480072 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-68b0f3cfa17644449993f63affcbe2be on tablet 68b0f3cfa17644449993f63affcbe2be with peer 83659c8dbbcd4142abdbcb243eeec8d4
W20240502 08:18:20.488245 15042 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 134664us
I20240502 08:18:20.509070 17268 tablet_copy_client.cc:308] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:43799
I20240502 08:18:20.593681 15256 tablet_copy_service.cc:140] P 19d550ab72684bf8aa1382d9c348cd29: Received BeginTabletCopySession request for tablet df91c5f0a46c43d78f8f7aeef7281e14 from peer 382fd0c6daaf4eddb1c57557e9b716d1 ({username='slave'} at 127.8.68.197:55353)
I20240502 08:18:20.593886 15256 tablet_copy_service.cc:161] P 19d550ab72684bf8aa1382d9c348cd29: Beginning new tablet copy session on tablet df91c5f0a46c43d78f8f7aeef7281e14 from peer 382fd0c6daaf4eddb1c57557e9b716d1 at {username='slave'} at 127.8.68.197:55353: session id = 382fd0c6daaf4eddb1c57557e9b716d1-df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:18:20.596752 15256 tablet_copy_source_session.cc:205] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.598004 17268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df91c5f0a46c43d78f8f7aeef7281e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.605480 17268 tablet_copy_client.cc:783] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.605782 17268 tablet_copy_client.cc:647] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.615273 17268 tablet_copy_client.cc:523] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.619513 17268 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap starting.
I20240502 08:18:20.758725 14975 consensus_queue.cc:237] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:20.761591 15233 raft_consensus.cc:1261] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240502 08:18:20.762308 16988 consensus_queue.cc:1035] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240502 08:18:20.780277 16988 raft_consensus.cc:2932] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index -1 to 154, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:20.787375 15233 raft_consensus.cc:2932] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index -1 to 154, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:20.813031 14829 catalog_manager.cc:5484] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index -1 to 154, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:20.838460 16289 raft_consensus.cc:1203] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.138->[1.139-1.159] Dedup: 1.149->[1.150-1.159]
W20240502 08:18:20.956887 14911 consensus_peers.cc:485] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535): Couldn't send request to peer 7757b11bdda3443199e83298ba5c8118. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc254d1803141fbab91423631ab8855. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:21.231287 17268 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=159 overwritten=0 applied=159 ignored=0} inserts{seen=831 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:21.231855 17268 tablet_bootstrap.cc:492] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap complete.
I20240502 08:18:21.232590 17268 ts_tablet_manager.cc:1367] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent bootstrapping tablet: real 0.613s user 0.246s sys 0.006s
I20240502 08:18:21.233546 17268 raft_consensus.cc:348] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:21.236922 17268 raft_consensus.cc:729] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382fd0c6daaf4eddb1c57557e9b716d1, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.237717 17268 consensus_queue.cc:260] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 159, Last appended: 1.159, Last appended by leader: 159, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:21.279995 17268 ts_tablet_manager.cc:1398] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent starting tablet: real 0.047s user 0.012s sys 0.023s
I20240502 08:18:21.280913 15256 tablet_copy_service.cc:342] P 19d550ab72684bf8aa1382d9c348cd29: Request end of tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-df91c5f0a46c43d78f8f7aeef7281e14 received from {username='slave'} at 127.8.68.197:55353
I20240502 08:18:21.281083 15256 tablet_copy_service.cc:433] P 19d550ab72684bf8aa1382d9c348cd29: ending tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-df91c5f0a46c43d78f8f7aeef7281e14 on tablet df91c5f0a46c43d78f8f7aeef7281e14 with peer 382fd0c6daaf4eddb1c57557e9b716d1
I20240502 08:18:21.305562 17315 ts_tablet_manager.cc:902] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:21.351593 17315 tablet_copy_client.cc:308] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:21.398955 16421 raft_consensus.cc:1203] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.150->[1.151-1.162] Dedup: 1.159->[1.160-1.162]
I20240502 08:18:21.431120 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 4cc254d1803141fbab91423631ab8855 from peer 7757b11bdda3443199e83298ba5c8118 ({username='slave'} at 127.8.68.199:45167)
I20240502 08:18:21.431334 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 4cc254d1803141fbab91423631ab8855 from peer 7757b11bdda3443199e83298ba5c8118 at {username='slave'} at 127.8.68.199:45167: session id = 7757b11bdda3443199e83298ba5c8118-4cc254d1803141fbab91423631ab8855
I20240502 08:18:21.434306 14995 tablet_copy_source_session.cc:205] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:21.436607 17315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc254d1803141fbab91423631ab8855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.448904 17315 tablet_copy_client.cc:783] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:21.449309 17315 tablet_copy_client.cc:647] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:21.453754 17315 tablet_copy_client.cc:523] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:21.462527 17315 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Bootstrap starting.
I20240502 08:18:21.494971 15104 consensus_queue.cc:237] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.154, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:21.513180 15233 raft_consensus.cc:1261] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240502 08:18:21.513940 17269 consensus_queue.cc:1035] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 154, Time since last communication: 0.000s
I20240502 08:18:21.519923 17080 raft_consensus.cc:2932] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.155: config changed from index -1 to 155, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:21.524063 15232 raft_consensus.cc:2932] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index -1 to 155, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:21.532650 14828 catalog_manager.cc:5484] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: config changed from index -1 to 155, VOTER a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
W20240502 08:18:21.554651 15040 consensus_peers.cc:485] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 -> Peer a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198:34645): Couldn't send request to peer a8cf9c3ffc34440594482a6718b596e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9acb74154404bd8829cc19db214068a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:21.962030 17315 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Bootstrap replayed 1/1 log segments. Stats: ops{read=156 overwritten=0 applied=156 ignored=0} inserts{seen=787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:21.965910 17361 ts_tablet_manager.cc:902] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Initiating tablet copy from peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:21.989229 17315 tablet_bootstrap.cc:492] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Bootstrap complete.
I20240502 08:18:21.990989 17315 ts_tablet_manager.cc:1367] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Time spent bootstrapping tablet: real 0.529s user 0.228s sys 0.007s
I20240502 08:18:21.994769 17315 raft_consensus.cc:348] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:22.015377 17315 raft_consensus.cc:729] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7757b11bdda3443199e83298ba5c8118, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.018560 17315 consensus_queue.cc:260] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.156, Last appended by leader: 156, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:22.103279 17361 tablet_copy_client.cc:308] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:44789
I20240502 08:18:22.116122 17315 ts_tablet_manager.cc:1398] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118: Time spent starting tablet: real 0.122s user 0.004s sys 0.010s
I20240502 08:18:22.117847 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 7757b11bdda3443199e83298ba5c8118-4cc254d1803141fbab91423631ab8855 received from {username='slave'} at 127.8.68.199:45167
I20240502 08:18:22.118021 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 7757b11bdda3443199e83298ba5c8118-4cc254d1803141fbab91423631ab8855 on tablet 4cc254d1803141fbab91423631ab8855 with peer 7757b11bdda3443199e83298ba5c8118
I20240502 08:18:22.209560 15126 tablet_copy_service.cc:140] P 7fe0e31240d145a28e75262f706ebb89: Received BeginTabletCopySession request for tablet f9acb74154404bd8829cc19db214068a from peer a8cf9c3ffc34440594482a6718b596e5 ({username='slave'} at 127.8.68.198:57383)
I20240502 08:18:22.214140 15126 tablet_copy_service.cc:161] P 7fe0e31240d145a28e75262f706ebb89: Beginning new tablet copy session on tablet f9acb74154404bd8829cc19db214068a from peer a8cf9c3ffc34440594482a6718b596e5 at {username='slave'} at 127.8.68.198:57383: session id = a8cf9c3ffc34440594482a6718b596e5-f9acb74154404bd8829cc19db214068a
I20240502 08:18:22.217804 15126 tablet_copy_source_session.cc:205] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:22.219362 17361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9acb74154404bd8829cc19db214068a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.224288 14975 consensus_queue.cc:237] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:22.227051 15103 raft_consensus.cc:1261] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
W20240502 08:18:22.227181 14908 consensus_peers.cc:485] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799): Couldn't send request to peer 83659c8dbbcd4142abdbcb243eeec8d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9738c0a921014b7d9b8c9a1be4cbb433. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.227705 16988 consensus_queue.cc:1035] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s
I20240502 08:18:22.229408 17361 tablet_copy_client.cc:783] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:22.229697 17361 tablet_copy_client.cc:647] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:22.233702 16129 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index -1 to 141, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:22.246672 15103 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:22.258762 17361 tablet_copy_client.cc:523] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:22.262530 17361 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Bootstrap starting.
I20240502 08:18:22.265640 14828 catalog_manager.cc:5484] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: config changed from index -1 to 141, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:22.365671 16683 raft_consensus.cc:1203] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.153->[1.154-1.173] Dedup: 1.156->[1.157-1.173]
I20240502 08:18:22.763123 17405 ts_tablet_manager.cc:902] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:22.927472 17405 tablet_copy_client.cc:308] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:22.930256 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 9738c0a921014b7d9b8c9a1be4cbb433 from peer 83659c8dbbcd4142abdbcb243eeec8d4 ({username='slave'} at 127.8.68.196:60007)
I20240502 08:18:22.930466 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 9738c0a921014b7d9b8c9a1be4cbb433 from peer 83659c8dbbcd4142abdbcb243eeec8d4 at {username='slave'} at 127.8.68.196:60007: session id = 83659c8dbbcd4142abdbcb243eeec8d4-9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:18:22.933607 14995 tablet_copy_source_session.cc:205] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:22.935071 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9738c0a921014b7d9b8c9a1be4cbb433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.942067 17405 tablet_copy_client.cc:783] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:22.942413 17405 tablet_copy_client.cc:647] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:22.952839 17405 tablet_copy_client.cc:523] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:22.957623 17405 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:18:23.038758 14975 consensus_queue.cc:237] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 1.193, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
W20240502 08:18:23.042691 14908 consensus_peers.cc:485] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747): Couldn't send request to peer 382fd0c6daaf4eddb1c57557e9b716d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cadca3ac819042f0bee9f3251283a6f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:23.062294 15102 raft_consensus.cc:1261] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 193. Preceding OpId from leader: term: 1 index: 194. (index mismatch)
I20240502 08:18:23.063030 17348 consensus_queue.cc:1035] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I20240502 08:18:23.065867 17348 raft_consensus.cc:2932] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.194: config changed from index -1 to 194, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:23.073443 15103 raft_consensus.cc:2932] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Committing config change with OpId 1.194: config changed from index -1 to 194, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:23.081317 14828 catalog_manager.cc:5484] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: config changed from index -1 to 194, VOTER 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:23.365760 17361 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=164 overwritten=0 applied=164 ignored=0} inserts{seen=836 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.376587 17361 tablet_bootstrap.cc:492] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Bootstrap complete.
I20240502 08:18:23.395730 17361 ts_tablet_manager.cc:1367] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Time spent bootstrapping tablet: real 1.133s user 0.268s sys 0.026s
I20240502 08:18:23.397516 17361 raft_consensus.cc:348] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:23.397773 17361 raft_consensus.cc:729] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.399406 17361 consensus_queue.cc:260] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:23.404609 17361 ts_tablet_manager.cc:1398] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:18:23.469369 15126 tablet_copy_service.cc:342] P 7fe0e31240d145a28e75262f706ebb89: Request end of tablet copy session a8cf9c3ffc34440594482a6718b596e5-f9acb74154404bd8829cc19db214068a received from {username='slave'} at 127.8.68.198:57383
I20240502 08:18:23.469581 15126 tablet_copy_service.cc:433] P 7fe0e31240d145a28e75262f706ebb89: ending tablet copy session a8cf9c3ffc34440594482a6718b596e5-f9acb74154404bd8829cc19db214068a on tablet f9acb74154404bd8829cc19db214068a with peer a8cf9c3ffc34440594482a6718b596e5
I20240502 08:18:23.574193 17427 ts_tablet_manager.cc:902] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:18:23.751948 17405 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=151 ignored=0} inserts{seen=836 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.752482 17405 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap complete.
I20240502 08:18:23.752897 17405 ts_tablet_manager.cc:1367] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 0.795s user 0.216s sys 0.025s
I20240502 08:18:23.753758 17405 raft_consensus.cc:348] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:23.754094 17405 raft_consensus.cc:729] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.754436 17405 consensus_queue.cc:260] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 1.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:23.848965 17427 tablet_copy_client.cc:308] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:18:23.851429 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet cadca3ac819042f0bee9f3251283a6f3 from peer 382fd0c6daaf4eddb1c57557e9b716d1 ({username='slave'} at 127.8.68.197:56427)
I20240502 08:18:23.851666 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet cadca3ac819042f0bee9f3251283a6f3 from peer 382fd0c6daaf4eddb1c57557e9b716d1 at {username='slave'} at 127.8.68.197:56427: session id = 382fd0c6daaf4eddb1c57557e9b716d1-cadca3ac819042f0bee9f3251283a6f3
I20240502 08:18:23.856829 17405 ts_tablet_manager.cc:1398] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.104s user 0.017s sys 0.035s
I20240502 08:18:23.858076 14994 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-9738c0a921014b7d9b8c9a1be4cbb433 received from {username='slave'} at 127.8.68.196:60007
I20240502 08:18:23.858239 14994 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-9738c0a921014b7d9b8c9a1be4cbb433 on tablet 9738c0a921014b7d9b8c9a1be4cbb433 with peer 83659c8dbbcd4142abdbcb243eeec8d4
I20240502 08:18:23.859088 16289 raft_consensus.cc:1203] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.140->[1.141-1.155] Dedup: 1.151->[1.152-1.155]
I20240502 08:18:23.864279 14995 tablet_copy_source_session.cc:205] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.869475 17427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadca3ac819042f0bee9f3251283a6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.885879 17427 tablet_copy_client.cc:783] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.886271 17427 tablet_copy_client.cc:647] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.937142 15235 consensus_queue.cc:237] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 1.184, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
W20240502 08:18:23.944588 15039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 44351.6us
W20240502 08:18:23.954001 15172 consensus_peers.cc:485] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535): Couldn't send request to peer 7757b11bdda3443199e83298ba5c8118. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e685b7cf44341c5b9e89360d8488338. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:23.954360 14972 raft_consensus.cc:1261] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 184. Preceding OpId from leader: term: 1 index: 185. (index mismatch)
I20240502 08:18:23.955192 17312 consensus_queue.cc:1035] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I20240502 08:18:23.958276 17312 raft_consensus.cc:2932] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.185: config changed from index -1 to 185, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:23.963887 14972 raft_consensus.cc:2932] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Committing config change with OpId 1.185: config changed from index -1 to 185, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:23.970041 16551 raft_consensus.cc:1203] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.154->[1.155-1.185] Dedup: 1.164->[1.165-1.185]
I20240502 08:18:23.972626 14829 catalog_manager.cc:5484] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index -1 to 185, VOTER 7757b11bdda3443199e83298ba5c8118 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:24.036610 17427 tablet_copy_client.cc:523] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:24.040479 17427 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap starting.
W20240502 08:18:24.079375 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 86208.3us
I20240502 08:18:24.539441 17474 ts_tablet_manager.cc:902] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Initiating tablet copy from peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:18:24.594794 17427 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=200 overwritten=0 applied=200 ignored=0} inserts{seen=831 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:24.595667 17427 tablet_bootstrap.cc:492] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Bootstrap complete.
W20240502 08:18:24.600557 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 60719.9us
I20240502 08:18:24.605114 17427 ts_tablet_manager.cc:1367] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent bootstrapping tablet: real 0.565s user 0.271s sys 0.022s
I20240502 08:18:24.606330 17427 raft_consensus.cc:348] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:24.606720 17427 raft_consensus.cc:729] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382fd0c6daaf4eddb1c57557e9b716d1, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.607129 17427 consensus_queue.cc:260] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.200, Last appended by leader: 200, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:24.628444 17427 ts_tablet_manager.cc:1398] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20240502 08:18:24.629200 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-cadca3ac819042f0bee9f3251283a6f3 received from {username='slave'} at 127.8.68.197:56427
I20240502 08:18:24.629369 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 382fd0c6daaf4eddb1c57557e9b716d1-cadca3ac819042f0bee9f3251283a6f3 on tablet cadca3ac819042f0bee9f3251283a6f3 with peer 382fd0c6daaf4eddb1c57557e9b716d1
I20240502 08:18:24.674253 16420 raft_consensus.cc:1203] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.193->[1.194-1.204] Dedup: 1.200->[1.201-1.204]
I20240502 08:18:24.704123 17474 tablet_copy_client.cc:308] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:43799
I20240502 08:18:24.711416 15256 tablet_copy_service.cc:140] P 19d550ab72684bf8aa1382d9c348cd29: Received BeginTabletCopySession request for tablet 8e685b7cf44341c5b9e89360d8488338 from peer 7757b11bdda3443199e83298ba5c8118 ({username='slave'} at 127.8.68.199:33267)
I20240502 08:18:24.711656 15256 tablet_copy_service.cc:161] P 19d550ab72684bf8aa1382d9c348cd29: Beginning new tablet copy session on tablet 8e685b7cf44341c5b9e89360d8488338 from peer 7757b11bdda3443199e83298ba5c8118 at {username='slave'} at 127.8.68.199:33267: session id = 7757b11bdda3443199e83298ba5c8118-8e685b7cf44341c5b9e89360d8488338
I20240502 08:18:24.714597 15256 tablet_copy_source_session.cc:205] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:24.717476 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e685b7cf44341c5b9e89360d8488338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.738193 17474 tablet_copy_client.cc:783] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:24.738775 17474 tablet_copy_client.cc:647] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:24.742658 17474 tablet_copy_client.cc:523] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:24.746569 17474 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Bootstrap starting.
W20240502 08:18:25.251581 15039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 56643us
I20240502 08:18:25.565848 15102 consensus_queue.cc:237] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:25.571815 15232 raft_consensus.cc:1261] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20240502 08:18:25.583333 17345 consensus_queue.cc:1035] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 08:18:25.596331 17079 raft_consensus.cc:2932] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index -1 to 220, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
W20240502 08:18:25.596798 15039 consensus_peers.cc:485] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 -> Peer 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799): Couldn't send request to peer 83659c8dbbcd4142abdbcb243eeec8d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73acd88beabf42679643daf4e62dea3b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.615676 15234 raft_consensus.cc:2932] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index -1 to 220, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:25.616197 14829 catalog_manager.cc:5484] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: config changed from index -1 to 220, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:25.835777 17474 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Bootstrap replayed 1/1 log segments. Stats: ops{read=192 overwritten=0 applied=192 ignored=0} inserts{seen=823 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.844616 17474 tablet_bootstrap.cc:492] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Bootstrap complete.
I20240502 08:18:25.845726 17474 ts_tablet_manager.cc:1367] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Time spent bootstrapping tablet: real 1.099s user 0.326s sys 0.092s
I20240502 08:18:25.846552 17474 raft_consensus.cc:348] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:25.846863 17474 raft_consensus.cc:729] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7757b11bdda3443199e83298ba5c8118, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.847127 17474 consensus_queue.cc:260] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 192, Last appended: 1.192, Last appended by leader: 192, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:25.908435 17474 ts_tablet_manager.cc:1398] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118: Time spent starting tablet: real 0.062s user 0.029s sys 0.031s
I20240502 08:18:25.911852 15256 tablet_copy_service.cc:342] P 19d550ab72684bf8aa1382d9c348cd29: Request end of tablet copy session 7757b11bdda3443199e83298ba5c8118-8e685b7cf44341c5b9e89360d8488338 received from {username='slave'} at 127.8.68.199:33267
I20240502 08:18:25.912027 15256 tablet_copy_service.cc:433] P 19d550ab72684bf8aa1382d9c348cd29: ending tablet copy session 7757b11bdda3443199e83298ba5c8118-8e685b7cf44341c5b9e89360d8488338 on tablet 8e685b7cf44341c5b9e89360d8488338 with peer 7757b11bdda3443199e83298ba5c8118
I20240502 08:18:25.946602 16682 raft_consensus.cc:1203] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.184->[1.185-1.207] Dedup: 1.192->[1.193-1.207]
I20240502 08:18:26.084969 17538 ts_tablet_manager.cc:902] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Initiating tablet copy from peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:26.155318 17538 tablet_copy_client.cc:308] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:44789
I20240502 08:18:26.157299 15126 tablet_copy_service.cc:140] P 7fe0e31240d145a28e75262f706ebb89: Received BeginTabletCopySession request for tablet 73acd88beabf42679643daf4e62dea3b from peer 83659c8dbbcd4142abdbcb243eeec8d4 ({username='slave'} at 127.8.68.196:33365)
I20240502 08:18:26.157508 15126 tablet_copy_service.cc:161] P 7fe0e31240d145a28e75262f706ebb89: Beginning new tablet copy session on tablet 73acd88beabf42679643daf4e62dea3b from peer 83659c8dbbcd4142abdbcb243eeec8d4 at {username='slave'} at 127.8.68.196:33365: session id = 83659c8dbbcd4142abdbcb243eeec8d4-73acd88beabf42679643daf4e62dea3b
I20240502 08:18:26.160377 15126 tablet_copy_source_session.cc:205] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.161811 17538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73acd88beabf42679643daf4e62dea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.214272 17538 tablet_copy_client.cc:783] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.219588 17538 tablet_copy_client.cc:647] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.267818 17538 tablet_copy_client.cc:523] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.295118 17538 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap starting.
I20240502 08:18:26.367944 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:26.394896 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.442345 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.488075 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:26.545255 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:26.570603 16289 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.000s sys 0.009s
W20240502 08:18:26.570991 16289 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.582s user 0.000s sys 0.009s
W20240502 08:18:26.600893 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 44777us
W20240502 08:18:26.601706 15169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 45541.2us
I20240502 08:18:26.647154 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.706213 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:26.747124 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 45387.5us
W20240502 08:18:26.897398 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 68583.9us
W20240502 08:18:26.899468 15039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 53579.2us
W20240502 08:18:27.359498 16484 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 78063.2us
I20240502 08:18:27.630523 17538 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=225 overwritten=0 applied=224 ignored=0} inserts{seen=896 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:27.631227 17538 tablet_bootstrap.cc:492] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Bootstrap complete.
I20240502 08:18:27.631666 17538 ts_tablet_manager.cc:1367] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent bootstrapping tablet: real 1.337s user 0.350s sys 0.019s
I20240502 08:18:27.632614 17538 raft_consensus.cc:348] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:27.634838 17538 raft_consensus.cc:729] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.635172 17538 consensus_queue.cc:260] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 224, Last appended: 1.225, Last appended by leader: 225, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:27.653302 17538 ts_tablet_manager.cc:1398] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4: Time spent starting tablet: real 0.021s user 0.002s sys 0.006s
I20240502 08:18:27.657771 15126 tablet_copy_service.cc:342] P 7fe0e31240d145a28e75262f706ebb89: Request end of tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-73acd88beabf42679643daf4e62dea3b received from {username='slave'} at 127.8.68.196:33365
I20240502 08:18:27.657968 15126 tablet_copy_service.cc:433] P 7fe0e31240d145a28e75262f706ebb89: ending tablet copy session 83659c8dbbcd4142abdbcb243eeec8d4-73acd88beabf42679643daf4e62dea3b on tablet 73acd88beabf42679643daf4e62dea3b with peer 83659c8dbbcd4142abdbcb243eeec8d4
I20240502 08:18:27.809834 16288 raft_consensus.cc:1203] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.219->[1.220-1.243] Dedup: 1.225->[1.226-1.243]
W20240502 08:18:28.245529 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 45135.5us
W20240502 08:18:28.796557 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 81726.5us
W20240502 08:18:28.817724 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 200102us
W20240502 08:18:29.208317 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 60724.8us
I20240502 08:18:29.682462 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:29.701069 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:18:29.802652 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 161937us
I20240502 08:18:29.811259 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.962982 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.998296 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:30.043599 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:30.060029 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 65842.4us
W20240502 08:18:30.251665 14910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 59914.9us
I20240502 08:18:30.366421 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:31.061169 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 52360.5us
I20240502 08:18:31.130934 14972 consensus_queue.cc:237] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.273, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:31.145720 16419 raft_consensus.cc:1261] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 274. (index mismatch)
I20240502 08:18:31.152511 16129 consensus_queue.cc:1035] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20240502 08:18:31.174134 16129 raft_consensus.cc:2932] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.274: config changed from index 194 to 274, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:31.185330 16419 raft_consensus.cc:2932] T cadca3ac819042f0bee9f3251283a6f3 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Committing config change with OpId 1.274: config changed from index 194 to 274, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:31.189438 14829 catalog_manager.cc:5484] T cadca3ac819042f0bee9f3251283a6f3 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index 194 to 274, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:31.450099 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet cadca3ac819042f0bee9f3251283a6f3 with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 274) from {username='slave'} at 127.0.0.1:43560
I20240502 08:18:31.477798 17738 tablet_replica.cc:330] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:18:31.480015 17738 raft_consensus.cc:2229] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:31.480361 17738 raft_consensus.cc:2258] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.588306 17738 ts_tablet_manager.cc:1875] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:31.631431 17738 ts_tablet_manager.cc:1888] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.273
I20240502 08:18:31.632122 17738 log.cc:1196] T cadca3ac819042f0bee9f3251283a6f3 P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/cadca3ac819042f0bee9f3251283a6f3
I20240502 08:18:31.633661 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet cadca3ac819042f0bee9f3251283a6f3 (table rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]) successfully deleted
W20240502 08:18:31.707363 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 89654.2us
W20240502 08:18:32.556293 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 63973.7us
I20240502 08:18:32.622749 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:32.628235 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:32.832062 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:32.858507 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:32.941326 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:32.985877 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:32.999974 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:33.129004 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 52625us
W20240502 08:18:33.200572 15169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 60532.9us
W20240502 08:18:33.201273 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 65745.2us
I20240502 08:18:34.179122 14972 consensus_queue.cc:237] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:34.185546 16289 raft_consensus.cc:1261] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 1 index: 266. (index mismatch)
I20240502 08:18:34.194114 17591 consensus_queue.cc:1035] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20240502 08:18:34.233968 17691 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.266: config changed from index 141 to 266, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:34.253626 14828 catalog_manager.cc:5484] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index 141 to 266, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:34.259697 16289 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Committing config change with OpId 1.266: config changed from index 141 to 266, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:34.534595 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 9738c0a921014b7d9b8c9a1be4cbb433 with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 266) from {username='slave'} at 127.0.0.1:43560
I20240502 08:18:34.569109 17840 tablet_replica.cc:330] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:18:34.570739 17840 raft_consensus.cc:2229] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:34.572834 17840 raft_consensus.cc:2258] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:34.574733 17840 ts_tablet_manager.cc:1875] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:34.612812 17840 ts_tablet_manager.cc:1888] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.265
I20240502 08:18:34.613070 17840 log.cc:1196] T 9738c0a921014b7d9b8c9a1be4cbb433 P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:18:34.614022 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet 9738c0a921014b7d9b8c9a1be4cbb433 (table rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]) successfully deleted
W20240502 08:18:35.332962 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 66644.1us
I20240502 08:18:35.770004 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:35.785830 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:35.824018 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 44054.7us
I20240502 08:18:35.862406 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:35.889499 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 53458.9us
I20240502 08:18:35.902727 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:35.924737 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:35.940819 14955 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:36.052332 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 54051.7us
I20240502 08:18:36.061774 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:36.899968 14972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cc254d1803141fbab91423631ab8855"
dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35094
I20240502 08:18:36.904345 14972 raft_consensus.cc:595] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:36.948830 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:36.954586 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:36.962302 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:36.972851 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:36.982638 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:36.990227 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.004590 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.026386 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.042690 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.056174 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.070487 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.091045 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.110548 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.136381 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.181105 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.208320 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.229653 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.257342 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.278856 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.300673 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.323137 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
W20240502 08:18:37.348455 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
I20240502 08:18:37.362015 17904 raft_consensus.cc:979] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: : Instructing follower 19d550ab72684bf8aa1382d9c348cd29 to start an election
I20240502 08:18:37.362251 17903 raft_consensus.cc:1067] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Signalling peer 19d550ab72684bf8aa1382d9c348cd29 to start an election
I20240502 08:18:37.363854 15234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cc254d1803141fbab91423631ab8855"
dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
from {username='slave'} at 127.8.68.193:55721
I20240502 08:18:37.364235 15234 raft_consensus.cc:482] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:37.373020 15234 raft_consensus.cc:3037] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:37.377862 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Service unavailable: leader transfer in progress
I20240502 08:18:37.382264 15234 raft_consensus.cc:504] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:37.385377 16682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc254d1803141fbab91423631ab8855" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 380 } } ignore_live_leader: true dest_uuid: "7757b11bdda3443199e83298ba5c8118"
I20240502 08:18:37.385788 16682 raft_consensus.cc:3037] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:37.387554 15234 leader_election.cc:288] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 2 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895), 7757b11bdda3443199e83298ba5c8118 (127.8.68.199:34535)
I20240502 08:18:37.395066 16682 raft_consensus.cc:2454] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 2.
I20240502 08:18:37.395712 16684 raft_consensus.cc:1226] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Rejecting Update request from peer 97037d47bf1249d2b5932b6553b907d4 for earlier term 1. Current term is 2. Ops: []
I20240502 08:18:37.397310 17772 consensus_queue.cc:1046] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: INVALID_TERM, Last received: 1.380, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240502 08:18:37.397634 17772 raft_consensus.cc:3032] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:37.397753 17772 raft_consensus.cc:729] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:37.400044 15172 leader_election.cc:302] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7757b11bdda3443199e83298ba5c8118; no voters:
I20240502 08:18:37.400372 17909 raft_consensus.cc:2781] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:37.400509 17909 raft_consensus.cc:686] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:37.398030 17772 consensus_queue.cc:260] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 380, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:37.400837 17772 raft_consensus.cc:3037] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:37.400805 17909 consensus_queue.cc:237] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 380, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:37.404596 14828 catalog_manager.cc:5484] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 1 to 2, leader changed from 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
W20240502 08:18:37.413365 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58010: Illegal state: replica 97037d47bf1249d2b5932b6553b907d4 is not leader of this config: current role FOLLOWER
I20240502 08:18:37.431182 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc254d1803141fbab91423631ab8855" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 380 } } ignore_live_leader: true dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:18:37.435070 14975 raft_consensus.cc:2454] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 2.
I20240502 08:18:37.683632 16682 raft_consensus.cc:1261] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 2 index: 382. (index mismatch)
I20240502 08:18:37.684443 17908 consensus_queue.cc:1035] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240502 08:18:37.714219 14975 raft_consensus.cc:1261] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 2 index: 382. (index mismatch)
I20240502 08:18:37.726085 17909 consensus_queue.cc:1035] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
W20240502 08:18:37.824611 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 99002.2us
I20240502 08:18:37.847972 17939 mvcc.cc:205] Tried to move back new op lower bound from 7023156910826082304 to 7023156909678755840. Current Snapshot: MvccSnapshot[applied={T|T < 7023156910826082304 or (T in {7023156910826082304})}]
I20240502 08:18:38.059201 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.063424 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:38.094969 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:38.159205 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 53288us
I20240502 08:18:38.233075 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.239722 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:38.241922 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:18:38.285883 15042 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 85343.4us
I20240502 08:18:38.436336 14955 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 08:18:38.475960 14910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 98684.1us
W20240502 08:18:39.080148 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 48457.7us
W20240502 08:18:39.206645 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 66650.3us
I20240502 08:18:39.216413 15235 consensus_queue.cc:237] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 2.398, Last appended by leader: 380, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:39.251600 16682 raft_consensus.cc:1261] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 2 index: 398. Preceding OpId from leader: term: 2 index: 399. (index mismatch)
I20240502 08:18:39.316920 17907 consensus_queue.cc:1035] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 397, Time since last communication: 0.000s
I20240502 08:18:39.405529 18017 raft_consensus.cc:2932] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 LEADER]: Committing config change with OpId 2.399: config changed from index 154 to 399, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:39.436275 14828 catalog_manager.cc:5484] T 4cc254d1803141fbab91423631ab8855 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 154 to 399, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:39.441088 16683 raft_consensus.cc:2932] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Committing config change with OpId 2.399: config changed from index 154 to 399, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:39.590402 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc254d1803141fbab91423631ab8855 with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 399) from {username='slave'} at 127.0.0.1:58056
I20240502 08:18:39.620445 18047 tablet_replica.cc:330] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:18:39.621057 18047 raft_consensus.cc:2229] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:39.621381 18047 raft_consensus.cc:2258] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:39.660912 18047 ts_tablet_manager.cc:1875] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:39.700577 18047 ts_tablet_manager.cc:1888] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.397
I20240502 08:18:39.701221 18047 log.cc:1196] T 4cc254d1803141fbab91423631ab8855 P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cc254d1803141fbab91423631ab8855
I20240502 08:18:39.708894 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet 4cc254d1803141fbab91423631ab8855 (table rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]) successfully deleted
W20240502 08:18:40.826171 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 44139.8us
I20240502 08:18:41.240346 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.248767 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.317353 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:41.412899 14955 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:18:41.449776 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:41.513286 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.553470 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:18:42.596189 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 60335us
I20240502 08:18:42.622305 15236 consensus_queue.cc:237] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 1.394, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:42.624168 16683 raft_consensus.cc:1261] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 394. Preceding OpId from leader: term: 1 index: 395. (index mismatch)
I20240502 08:18:42.624887 18017 consensus_queue.cc:1035] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I20240502 08:18:42.628641 18017 raft_consensus.cc:2932] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.395: config changed from index 185 to 395, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:42.631265 16683 raft_consensus.cc:2932] T 8e685b7cf44341c5b9e89360d8488338 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Committing config change with OpId 1.395: config changed from index 185 to 395, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:42.647989 14827 catalog_manager.cc:5484] T 8e685b7cf44341c5b9e89360d8488338 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 185 to 395, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:42.749207 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e685b7cf44341c5b9e89360d8488338 with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 395) from {username='slave'} at 127.0.0.1:58056
I20240502 08:18:42.791715 18166 tablet_replica.cc:330] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:18:42.792196 18166 raft_consensus.cc:2229] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:42.792856 18166 raft_consensus.cc:2258] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:42.796489 18166 ts_tablet_manager.cc:1875] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:42.804124 18166 ts_tablet_manager.cc:1888] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.394
I20240502 08:18:42.804344 18166 log.cc:1196] T 8e685b7cf44341c5b9e89360d8488338 P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/8e685b7cf44341c5b9e89360d8488338
I20240502 08:18:42.805557 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet 8e685b7cf44341c5b9e89360d8488338 (table rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]) successfully deleted
I20240502 08:18:43.760181 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:18:43.783617 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 43532.4us
I20240502 08:18:43.785773 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:43.891976 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:43.902209 16268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:43.936245 14955 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:18:43.952970 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:43.982729 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:18:44.057382 14909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:34645, user_credentials={real_user=slave}} blocked reactor thread for 92590.5us
W20240502 08:18:44.115396 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 77580.2us
I20240502 08:18:44.782820 14975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be"
dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43772
I20240502 08:18:44.783661 14975 raft_consensus.cc:595] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:44.866724 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.873965 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.881816 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.888373 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.895918 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.906895 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.915367 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.933774 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.957055 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.973933 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:44.994179 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:45.015192 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:45.037024 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:45.052996 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:45.073796 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
W20240502 08:18:45.093917 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
I20240502 08:18:45.102423 18116 raft_consensus.cc:979] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: : Instructing follower 83659c8dbbcd4142abdbcb243eeec8d4 to start an election
I20240502 08:18:45.102620 18116 raft_consensus.cc:1067] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Signalling peer 83659c8dbbcd4142abdbcb243eeec8d4 to start an election
I20240502 08:18:45.109148 16285 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be"
dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
from {username='slave'} at 127.8.68.193:49037
I20240502 08:18:45.109525 16285 raft_consensus.cc:482] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:45.109692 16285 raft_consensus.cc:3037] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:45.113201 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Service unavailable: leader transfer in progress
I20240502 08:18:45.120682 16285 raft_consensus.cc:504] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:45.122215 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be" candidate_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 465 } } ignore_live_leader: true dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:45.122222 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b0f3cfa17644449993f63affcbe2be" candidate_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 465 } } ignore_live_leader: true dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:18:45.122406 15106 raft_consensus.cc:3037] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:45.122438 14975 raft_consensus.cc:3032] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:45.122537 14975 raft_consensus.cc:729] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:18:45.122795 14975 consensus_queue.cc:260] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 465, Committed index: 465, Last appended: 1.465, Last appended by leader: 465, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:45.123118 14975 raft_consensus.cc:3037] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:45.125149 15106 raft_consensus.cc:2454] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83659c8dbbcd4142abdbcb243eeec8d4 in term 2.
I20240502 08:18:45.121824 16285 leader_election.cc:288] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [CANDIDATE]: Term 2 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895), 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:45.125869 16223 leader_election.cc:302] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fe0e31240d145a28e75262f706ebb89, 83659c8dbbcd4142abdbcb243eeec8d4; no voters:
I20240502 08:18:45.133361 14975 raft_consensus.cc:2454] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83659c8dbbcd4142abdbcb243eeec8d4 in term 2.
W20240502 08:18:45.140798 14935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Illegal state: replica 97037d47bf1249d2b5932b6553b907d4 is not leader of this config: current role FOLLOWER
I20240502 08:18:45.156236 18258 raft_consensus.cc:2781] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:45.223994 18258 raft_consensus.cc:686] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 LEADER]: Becoming Leader. State: Replica: 83659c8dbbcd4142abdbcb243eeec8d4, State: Running, Role: LEADER
I20240502 08:18:45.224714 18258 consensus_queue.cc:237] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 465, Committed index: 465, Last appended: 1.465, Last appended by leader: 465, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:45.231222 14827 catalog_manager.cc:5484] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 reported cstate change: term changed from 1 to 2, leader changed from 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) to 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196). New cstate: current_term: 2 leader_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
W20240502 08:18:45.234678 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48574: Illegal state: replica 7fe0e31240d145a28e75262f706ebb89 is not leader of this config: current role FOLLOWER
W20240502 08:18:45.275930 14933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57984: Illegal state: replica 97037d47bf1249d2b5932b6553b907d4 is not leader of this config: current role FOLLOWER
W20240502 08:18:45.298681 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48574: Illegal state: replica 7fe0e31240d145a28e75262f706ebb89 is not leader of this config: current role FOLLOWER
I20240502 08:18:45.485070 14975 raft_consensus.cc:1261] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Refusing update from remote peer 83659c8dbbcd4142abdbcb243eeec8d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 465. Preceding OpId from leader: term: 2 index: 467. (index mismatch)
I20240502 08:18:45.486042 15106 raft_consensus.cc:1261] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 83659c8dbbcd4142abdbcb243eeec8d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 465. Preceding OpId from leader: term: 2 index: 467. (index mismatch)
I20240502 08:18:45.489303 18259 consensus_queue.cc:1035] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 466, Last known committed idx: 465, Time since last communication: 0.000s
I20240502 08:18:45.533717 18235 mvcc.cc:205] Tried to move back new op lower bound from 7023156942773682176 to 7023156941735055360. Current Snapshot: MvccSnapshot[applied={T|T < 7023156942773682176 or (T in {7023156942773682176})}]
I20240502 08:18:45.557525 18258 consensus_queue.cc:1035] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 466, Last known committed idx: 465, Time since last communication: 0.000s
I20240502 08:18:46.123131 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:46.179587 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:46.245169 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:46.267632 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:46.309845 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:46.322369 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:46.556672 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:18:46.693717 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 121347us
W20240502 08:18:47.329896 16794 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 52077.1us
W20240502 08:18:47.428114 16223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 65045.9us
I20240502 08:18:47.694026 16285 consensus_queue.cc:237] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 505, Committed index: 505, Last appended: 2.505, Last appended by leader: 465, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:18:47.710853 15103 raft_consensus.cc:1261] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 83659c8dbbcd4142abdbcb243eeec8d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 505. Preceding OpId from leader: term: 2 index: 506. (index mismatch)
I20240502 08:18:47.713196 18345 consensus_queue.cc:1035] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 506, Last known committed idx: 505, Time since last communication: 0.000s
I20240502 08:18:47.724063 18258 raft_consensus.cc:2932] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 LEADER]: Committing config change with OpId 2.506: config changed from index 139 to 506, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:47.724581 15103 raft_consensus.cc:2932] T 68b0f3cfa17644449993f63affcbe2be P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Committing config change with OpId 2.506: config changed from index 139 to 506, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:47.734151 14834 catalog_manager.cc:5484] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 reported cstate change: config changed from index 139 to 506, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" committed_config { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:18:47.845618 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b0f3cfa17644449993f63affcbe2be with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 506) from {username='slave'} at 127.0.0.1:58056
I20240502 08:18:47.908931 18384 tablet_replica.cc:330] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:18:47.909168 18384 raft_consensus.cc:2229] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:47.909322 18384 pending_rounds.cc:62] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Trying to abort 1 pending ops.
I20240502 08:18:47.909406 18384 pending_rounds.cc:68] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Aborting op as it isn't in flight: id { term: 2 index: 505 } timestamp: 7023156951786303488 op_type: WRITE_OP write_request { tablet_id: "68b0f3cfa17644449993f63affcbe2be" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\253\032\234%\236N\344\305\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d381a93a4b1a51ae" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023156951752876032 authz_token { token_data: "\010\357\232\315\261\006\"3\n\005slave\022*\n 822e4752c7c24432937843b5ac7e12db\020\001\030\001 \001(\001" signature: "<redacted>""\222\222T\220\320\025\240\250\255P~\305{\'\252\214%\335\236)\n\026\261\351Iu\037\271\034\303o\254]y\361\246\224\322\233\361\332\364\014SM\212\350\267m\273\332\3713\007a/\314\354\316C\311O\335h" signing_key_seq_num: 0 } } request_id { client_id: "f61574e36fd9400ca91cef2c62e7752b" seq_no: 1525 first_incomplete_seq_no: 1525 attempt_no: 0 }
I20240502 08:18:47.910012 18384 raft_consensus.cc:2258] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:47.911590 18384 ts_tablet_manager.cc:1875] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:47.972142 18384 ts_tablet_manager.cc:1888] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.505
I20240502 08:18:47.972316 18384 log.cc:1196] T 68b0f3cfa17644449993f63affcbe2be P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/68b0f3cfa17644449993f63affcbe2be
I20240502 08:18:47.978878 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet 68b0f3cfa17644449993f63affcbe2be (table rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]) successfully deleted
I20240502 08:18:49.141186 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:49.293924 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:49.331895 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:49.352342 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:49.456617 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:49.463919 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:49.793300 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 92948.6us
I20240502 08:18:49.958235 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:50.429476 15232 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14"
dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49772
I20240502 08:18:50.430373 15232 raft_consensus.cc:595] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:50.450055 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:33895, user_credentials={real_user=slave}} blocked reactor thread for 102702us
I20240502 08:18:50.450409 18283 raft_consensus.cc:979] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: : Instructing follower 382fd0c6daaf4eddb1c57557e9b716d1 to start an election
I20240502 08:18:50.450526 18283 raft_consensus.cc:1067] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Signalling peer 382fd0c6daaf4eddb1c57557e9b716d1 to start an election
I20240502 08:18:50.454607 16418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14"
dest_uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
from {username='slave'} at 127.8.68.195:60691
I20240502 08:18:50.455036 16418 raft_consensus.cc:482] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:50.455371 16418 raft_consensus.cc:3037] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:50.458845 16418 raft_consensus.cc:504] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:50.461058 16418 leader_election.cc:288] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [CANDIDATE]: Term 2 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799), 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:18:50.464075 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14" candidate_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: true dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:18:50.464316 15103 raft_consensus.cc:3037] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:50.475154 15235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df91c5f0a46c43d78f8f7aeef7281e14" candidate_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: true dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:18:50.475533 15235 raft_consensus.cc:3032] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:50.475698 15235 raft_consensus.cc:729] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:18:50.476071 15235 consensus_queue.cc:260] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 477, Committed index: 477, Last appended: 1.477, Last appended by leader: 477, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:50.476583 15235 raft_consensus.cc:3037] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:50.467218 15103 raft_consensus.cc:2454] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382fd0c6daaf4eddb1c57557e9b716d1 in term 2.
I20240502 08:18:50.486296 16357 leader_election.cc:302] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382fd0c6daaf4eddb1c57557e9b716d1, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:18:50.491663 18470 raft_consensus.cc:2781] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:18:50.505172 15196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Illegal state: replica 19d550ab72684bf8aa1382d9c348cd29 is not leader of this config: current role FOLLOWER
I20240502 08:18:50.577219 15235 raft_consensus.cc:2454] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382fd0c6daaf4eddb1c57557e9b716d1 in term 2.
W20240502 08:18:50.607779 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48596: Illegal state: replica 7fe0e31240d145a28e75262f706ebb89 is not leader of this config: current role FOLLOWER
W20240502 08:18:50.617267 15196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Illegal state: replica 19d550ab72684bf8aa1382d9c348cd29 is not leader of this config: current role FOLLOWER
W20240502 08:18:50.626886 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48596: Illegal state: replica 7fe0e31240d145a28e75262f706ebb89 is not leader of this config: current role FOLLOWER
I20240502 08:18:50.688671 18470 raft_consensus.cc:686] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 LEADER]: Becoming Leader. State: Replica: 382fd0c6daaf4eddb1c57557e9b716d1, State: Running, Role: LEADER
I20240502 08:18:50.690301 18470 consensus_queue.cc:237] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 477, Committed index: 477, Last appended: 1.477, Last appended by leader: 477, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:50.704885 14833 catalog_manager.cc:5484] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: term changed from 1 to 2, leader changed from 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) to 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:50.892519 15102 raft_consensus.cc:1261] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 382fd0c6daaf4eddb1c57557e9b716d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 476. Preceding OpId from leader: term: 2 index: 479. (index mismatch)
I20240502 08:18:50.897040 18471 consensus_queue.cc:1035] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 478, Last known committed idx: 475, Time since last communication: 0.000s
I20240502 08:18:50.939553 15236 raft_consensus.cc:1261] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Refusing update from remote peer 382fd0c6daaf4eddb1c57557e9b716d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 477. Preceding OpId from leader: term: 2 index: 479. (index mismatch)
I20240502 08:18:50.954463 18471 consensus_queue.cc:1035] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 478, Last known committed idx: 477, Time since last communication: 0.000s
I20240502 08:18:51.542508 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:51.608342 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:51.615429 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:51.647475 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:51.666565 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:51.682634 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:51.716733 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:52.973541 16420 consensus_queue.cc:237] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 2.495, Last appended by leader: 477, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:18:52.978698 15102 raft_consensus.cc:1261] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 382fd0c6daaf4eddb1c57557e9b716d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 495. Preceding OpId from leader: term: 2 index: 496. (index mismatch)
I20240502 08:18:52.983142 18542 consensus_queue.cc:1035] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240502 08:18:52.993381 18542 raft_consensus.cc:2932] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 LEADER]: Committing config change with OpId 2.496: config changed from index 151 to 496, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:53.004274 14827 catalog_manager.cc:5484] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: config changed from index 151 to 496, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 2 leader_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" committed_config { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:53.012492 15103 raft_consensus.cc:2932] T df91c5f0a46c43d78f8f7aeef7281e14 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Committing config change with OpId 2.496: config changed from index 151 to 496, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:18:53.191109 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet df91c5f0a46c43d78f8f7aeef7281e14 with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 496) from {username='slave'} at 127.0.0.1:55452
I20240502 08:18:53.192132 18588 tablet_replica.cc:330] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:18:53.192775 18588 raft_consensus.cc:2229] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:53.193149 18588 raft_consensus.cc:2258] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:53.195137 18588 ts_tablet_manager.cc:1875] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:53.195791 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 67945us
I20240502 08:18:53.202802 18588 ts_tablet_manager.cc:1888] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.495
I20240502 08:18:53.203058 18588 log.cc:1196] T df91c5f0a46c43d78f8f7aeef7281e14 P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:18:53.204258 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet df91c5f0a46c43d78f8f7aeef7281e14 (table rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]) successfully deleted
W20240502 08:18:54.261955 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 46293.8us
I20240502 08:18:54.931677 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:54.946575 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:55.003827 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:55.023521 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:55.154721 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:18:55.185693 15041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 44695.5us
I20240502 08:18:55.263360 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:55.368868 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:55.724001 16484 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 101166us
W20240502 08:18:56.048516 14910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:43799, user_credentials={real_user=slave}} blocked reactor thread for 48117.1us
I20240502 08:18:56.090819 14972 consensus_queue.cc:237] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 776, Committed index: 776, Last appended: 1.776, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 777 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:18:56.094736 16550 raft_consensus.cc:1261] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 776. Preceding OpId from leader: term: 1 index: 777. (index mismatch)
I20240502 08:18:56.100466 18425 consensus_queue.cc:1035] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 777, Last known committed idx: 776, Time since last communication: 0.000s
I20240502 08:18:56.114888 18544 raft_consensus.cc:2932] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.777: config changed from index 114 to 777, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 777 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:56.121129 14833 catalog_manager.cc:5484] T 445454fd0e3d405d87fc7ef20929e862 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index 114 to 777, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 777 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:56.132866 16552 raft_consensus.cc:2932] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Committing config change with OpId 1.777: config changed from index 114 to 777, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 777 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:18:56.299000 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet 445454fd0e3d405d87fc7ef20929e862 with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 777) from {username='slave'} at 127.0.0.1:55452
I20240502 08:18:56.428004 18710 tablet_replica.cc:330] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:18:56.430132 18710 raft_consensus.cc:2229] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:56.430707 18710 raft_consensus.cc:2258] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:56.432461 18710 ts_tablet_manager.cc:1875] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:56.445444 18710 ts_tablet_manager.cc:1888] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.776
I20240502 08:18:56.445662 18710 log.cc:1196] T 445454fd0e3d405d87fc7ef20929e862 P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/445454fd0e3d405d87fc7ef20929e862
I20240502 08:18:56.447441 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet 445454fd0e3d405d87fc7ef20929e862 (table rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]) successfully deleted
I20240502 08:18:57.217248 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:57.271054 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:57.307783 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:57.315733 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:57.333283 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:57.360309 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:18:57.397022 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:58.457087 15235 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:18:58.476449 16684 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 1 index: 519. (index mismatch)
I20240502 08:18:58.525945 18763 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20240502 08:18:58.580227 18651 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.519: config changed from index 67 to 519, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:58.598986 14833 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 67 to 519, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:58.617084 16684 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Committing config change with OpId 1.519: config changed from index 67 to 519, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:18:58.634477 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f290f16fe044658ed9b6b9a0a71f08 with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 519) from {username='slave'} at 127.0.0.1:58056
I20240502 08:18:58.644152 18801 tablet_replica.cc:330] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:18:58.656723 18801 raft_consensus.cc:2229] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:58.657207 18801 raft_consensus.cc:2258] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:58.668629 18801 ts_tablet_manager.cc:1875] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:58.695757 18801 ts_tablet_manager.cc:1888] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.518
I20240502 08:18:58.696094 18801 log.cc:1196] T b2f290f16fe044658ed9b6b9a0a71f08 P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:18:58.697391 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet b2f290f16fe044658ed9b6b9a0a71f08 (table rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]) successfully deleted
W20240502 08:18:58.994354 14911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 73401.5us
I20240502 08:19:00.537640 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:00.750135 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:00.871948 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.880081 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:00.887977 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:00.888123 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:00.986447 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:01.951689 15106 consensus_queue.cc:237] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 639, Committed index: 639, Last appended: 1.639, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:19:01.953967 16551 raft_consensus.cc:1261] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 639. Preceding OpId from leader: term: 1 index: 640. (index mismatch)
I20240502 08:19:01.954631 18444 consensus_queue.cc:1035] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 640, Last known committed idx: 639, Time since last communication: 0.000s
I20240502 08:19:01.958024 18448 raft_consensus.cc:2932] T f9acb74154404bd8829cc19db214068a P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.640: config changed from index 155 to 640, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:01.958197 16551 raft_consensus.cc:2932] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Committing config change with OpId 1.640: config changed from index 155 to 640, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:01.965066 14833 catalog_manager.cc:5484] T f9acb74154404bd8829cc19db214068a P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: config changed from index 155 to 640, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:02.207408 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet f9acb74154404bd8829cc19db214068a with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 640) from {username='slave'} at 127.0.0.1:55452
I20240502 08:19:02.539490 18931 tablet_replica.cc:330] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:19:02.540520 18931 raft_consensus.cc:2229] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:02.540788 18931 pending_rounds.cc:62] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Trying to abort 1 pending ops.
I20240502 08:19:02.540962 18931 pending_rounds.cc:68] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Aborting op as it isn't in flight: id { term: 1 index: 639 } timestamp: 7023157010186563584 op_type: WRITE_OP write_request { tablet_id: "f9acb74154404bd8829cc19db214068a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\025(\016wE\205\237\252\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""00d3af78934a2110" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023157010122588160 authz_token { token_data: "\010\357\232\315\261\006\"3\n\005slave\022*\n f4c7e164dcda4ea1a90e063191d0d9f0\020\001\030\001 \001(\001" signature: "<redacted>""u{\215\204\217#\203\002w\321\035\211\376_u\266Q\003\267\003\004\237\310\'Q+[6\021\274\371\013+\202\034o\202D\337\276X\337~\3568\361\310\355\225D/\356>\311::t\355\247m\013\013\035\303" signing_key_seq_num: 0 } } request_id { client_id: "b6fd409391f245ceaad0ccd9b4c6b2ef" seq_no: 1826 first_incomplete_seq_no: 1826 attempt_no: 0 }
I20240502 08:19:02.545094 18931 raft_consensus.cc:2258] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:02.546571 18931 ts_tablet_manager.cc:1875] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:02.607753 18931 ts_tablet_manager.cc:1888] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.639
I20240502 08:19:02.607904 18931 log.cc:1196] T f9acb74154404bd8829cc19db214068a P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9acb74154404bd8829cc19db214068a
I20240502 08:19:02.608875 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet f9acb74154404bd8829cc19db214068a (table rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]) successfully deleted
I20240502 08:19:04.115429 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:04.125622 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:04.196857 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:04.236523 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:04.243468 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:04.457257 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:04.499547 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:19:04.530583 16484 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 51519.2us
W20240502 08:19:04.776260 15039 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 71412.4us
W20240502 08:19:05.272318 14909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:34645, user_credentials={real_user=slave}} blocked reactor thread for 68360.4us
I20240502 08:19:05.278674 14972 consensus_queue.cc:237] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 773, Committed index: 773, Last appended: 1.773, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:19:05.280656 16420 raft_consensus.cc:1261] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 773. Preceding OpId from leader: term: 1 index: 774. (index mismatch)
I20240502 08:19:05.281359 19041 consensus_queue.cc:1035] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 774, Last known committed idx: 773, Time since last communication: 0.000s
I20240502 08:19:05.286804 16420 raft_consensus.cc:2932] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Committing config change with OpId 1.774: config changed from index 66 to 774, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:05.286245 19033 raft_consensus.cc:2932] T 4c94e28f8f2b47cba77ed459c8e731d0 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.774: config changed from index 66 to 774, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:05.293988 14834 catalog_manager.cc:5484] T 4c94e28f8f2b47cba77ed459c8e731d0 P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: config changed from index 66 to 774, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:05.343521 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c94e28f8f2b47cba77ed459c8e731d0 with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 774) from {username='slave'} at 127.0.0.1:43560
I20240502 08:19:05.370193 19051 tablet_replica.cc:330] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:19:05.370553 19051 raft_consensus.cc:2229] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:05.370836 19051 raft_consensus.cc:2258] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:05.371934 19051 ts_tablet_manager.cc:1875] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:05.401912 19051 ts_tablet_manager.cc:1888] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.773
I20240502 08:19:05.405793 19051 log.cc:1196] T 4c94e28f8f2b47cba77ed459c8e731d0 P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:19:05.407375 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet 4c94e28f8f2b47cba77ed459c8e731d0 (table rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]) successfully deleted
I20240502 08:19:06.439108 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:06.469377 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:06.521597 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:06.548436 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:06.550803 15085 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:06.703572 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:06.721370 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:06.976207 15106 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3"
dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55914
I20240502 08:19:06.976516 15106 raft_consensus.cc:595] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:19:07.110690 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.118002 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.127331 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.138440 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.159816 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.172573 15065 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.195578 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.261476 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.272194 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.284379 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.296248 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.309422 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
W20240502 08:19:07.324337 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
I20240502 08:19:07.331235 19015 raft_consensus.cc:979] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: : Instructing follower 19d550ab72684bf8aa1382d9c348cd29 to start an election
I20240502 08:19:07.331439 19015 raft_consensus.cc:1067] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Signalling peer 19d550ab72684bf8aa1382d9c348cd29 to start an election
I20240502 08:19:07.333021 15234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3"
dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
from {username='slave'} at 127.8.68.194:58437
I20240502 08:19:07.333249 15234 raft_consensus.cc:482] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:07.333338 15234 raft_consensus.cc:3037] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:19:07.343719 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Service unavailable: leader transfer in progress
I20240502 08:19:07.343894 15234 raft_consensus.cc:504] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:19:07.352005 15234 leader_election.cc:288] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 2 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789), 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799)
I20240502 08:19:07.361864 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 874 } } ignore_live_leader: true dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:19:07.362098 15103 raft_consensus.cc:3032] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:07.362188 15103 raft_consensus.cc:729] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:19:07.362533 15103 consensus_queue.cc:260] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 874, Committed index: 874, Last appended: 1.874, Last appended by leader: 874, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:19:07.362998 15103 raft_consensus.cc:3037] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:07.366875 15103 raft_consensus.cc:2454] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 2.
W20240502 08:19:07.367938 15066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37554: Illegal state: replica 7fe0e31240d145a28e75262f706ebb89 is not leader of this config: current role FOLLOWER
I20240502 08:19:07.370368 15172 leader_election.cc:302] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d550ab72684bf8aa1382d9c348cd29, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:19:07.376030 19122 raft_consensus.cc:2781] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:07.376207 19122 raft_consensus.cc:686] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 LEADER]: Becoming Leader. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:19:07.376483 19122 consensus_queue.cc:237] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 874, Committed index: 874, Last appended: 1.874, Last appended by leader: 874, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:19:07.393000 14834 catalog_manager.cc:5484] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: term changed from 1 to 2, leader changed from 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) to 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:07.467056 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa2b2b96e134484a50edaac02dcc2a3" candidate_uuid: "19d550ab72684bf8aa1382d9c348cd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 874 } } ignore_live_leader: true dest_uuid: "83659c8dbbcd4142abdbcb243eeec8d4"
I20240502 08:19:07.467362 16289 raft_consensus.cc:3037] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:07.470739 16289 raft_consensus.cc:2454] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d550ab72684bf8aa1382d9c348cd29 in term 2.
I20240502 08:19:07.490872 15103 raft_consensus.cc:1261] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 874. Preceding OpId from leader: term: 2 index: 876. (index mismatch)
I20240502 08:19:07.492679 19122 consensus_queue.cc:1035] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 875, Last known committed idx: 874, Time since last communication: 0.000s
I20240502 08:19:07.537703 16284 raft_consensus.cc:1261] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 874. Preceding OpId from leader: term: 2 index: 876. (index mismatch)
I20240502 08:19:07.540223 19118 consensus_queue.cc:1035] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 875, Last known committed idx: 874, Time since last communication: 0.000s
W20240502 08:19:07.571062 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 62751.1us
I20240502 08:19:07.883198 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:07.883956 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:07.975438 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:08.020031 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:08.078905 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:08.105274 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:08.207067 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:08.942687 15234 consensus_queue.cc:237] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 895, Committed index: 895, Last appended: 2.895, Last appended by leader: 874, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 896 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:19:08.946520 16284 raft_consensus.cc:1261] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 2 index: 895. Preceding OpId from leader: term: 2 index: 897. (index mismatch)
I20240502 08:19:08.958997 18872 consensus_queue.cc:1035] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 896, Last known committed idx: 895, Time since last communication: 0.000s
I20240502 08:19:08.968134 18872 raft_consensus.cc:2932] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 LEADER]: Committing config change with OpId 2.896: config changed from index 93 to 896, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 896 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:08.969010 16286 raft_consensus.cc:2932] T 1aa2b2b96e134484a50edaac02dcc2a3 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 2 FOLLOWER]: Committing config change with OpId 2.896: config changed from index 93 to 896, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 896 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:08.976682 14834 catalog_manager.cc:5484] T 1aa2b2b96e134484a50edaac02dcc2a3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 93 to 896, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 896 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:09.009315 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa2b2b96e134484a50edaac02dcc2a3 with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 896) from {username='slave'} at 127.0.0.1:43560
I20240502 08:19:09.009930 19244 tablet_replica.cc:330] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:19:09.011303 19244 raft_consensus.cc:2229] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:09.011605 19244 raft_consensus.cc:2258] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:09.013121 19244 ts_tablet_manager.cc:1875] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:09.039419 19244 ts_tablet_manager.cc:1888] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.895
I20240502 08:19:09.039783 19244 log.cc:1196] T 1aa2b2b96e134484a50edaac02dcc2a3 P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:19:09.041153 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet 1aa2b2b96e134484a50edaac02dcc2a3 (table rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]) successfully deleted
I20240502 08:19:10.210227 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:10.237289 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:10.263398 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:19:10.390623 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:41747, user_credentials={real_user=slave}} blocked reactor thread for 44741.8us
W20240502 08:19:10.456460 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 47230.9us
I20240502 08:19:10.489180 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:10.496783 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:10.516289 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:10.654778 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:19:11.693464 16484 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 50689.7us
I20240502 08:19:11.721693 15234 consensus_queue.cc:237] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 903, Committed index: 903, Last appended: 1.903, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 904 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:19:11.723326 16684 raft_consensus.cc:1261] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 903. Preceding OpId from leader: term: 1 index: 904. (index mismatch)
I20240502 08:19:11.724087 19321 consensus_queue.cc:1035] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 904, Last known committed idx: 903, Time since last communication: 0.000s
I20240502 08:19:11.730860 19313 raft_consensus.cc:2932] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.904: config changed from index 123 to 904, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 904 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:19:11.736841 16680 raft_consensus.cc:2932] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Committing config change with OpId 1.904: config changed from index 123 to 904, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 904 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:19:11.778475 14827 catalog_manager.cc:5484] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 123 to 904, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 904 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } }
I20240502 08:19:11.907677 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7ca0fbedbe43e2a42fd849c233ded3 with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 904) from {username='slave'} at 127.0.0.1:43560
I20240502 08:19:11.908581 19356 tablet_replica.cc:330] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:19:11.908859 19356 raft_consensus.cc:2229] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:11.909058 19356 raft_consensus.cc:2258] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:11.910090 19356 ts_tablet_manager.cc:1875] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:11.924714 19356 ts_tablet_manager.cc:1888] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.903
I20240502 08:19:11.924937 19356 log.cc:1196] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:19:11.925992 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet 4e7ca0fbedbe43e2a42fd849c233ded3 (table rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]) successfully deleted
I20240502 08:19:12.950435 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:12.958485 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:13.003113 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:13.051440 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:13.112480 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:13.194638 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:13.239449 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:14.003782 14975 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 937, Committed index: 937, Last appended: 1.937, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 938 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:19:14.006433 16552 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 937. Preceding OpId from leader: term: 1 index: 938. (index mismatch)
I20240502 08:19:14.008653 19309 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 938, Last known committed idx: 937, Time since last communication: 0.000s
I20240502 08:19:14.013734 19136 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.938: config changed from index 72 to 938, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 938 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:14.019811 16550 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Committing config change with OpId 1.938: config changed from index 72 to 938, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 938 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:14.025063 14834 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: config changed from index 72 to 938, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 938 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } }
I20240502 08:19:14.136871 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet 4485a1686beb45d7ad56010d2a7c9fc9 with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 938) from {username='slave'} at 127.0.0.1:55452
I20240502 08:19:14.178763 19453 tablet_replica.cc:330] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:19:14.179118 19453 raft_consensus.cc:2229] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:14.179415 19453 raft_consensus.cc:2258] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:14.180507 19453 ts_tablet_manager.cc:1875] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:14.202029 19453 ts_tablet_manager.cc:1888] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.937
I20240502 08:19:14.202245 19453 log.cc:1196] T 4485a1686beb45d7ad56010d2a7c9fc9 P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:14.203281 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet 4485a1686beb45d7ad56010d2a7c9fc9 (table rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]) successfully deleted
I20240502 08:19:14.923974 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:14.945178 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:14.961578 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:15.006270 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:15.035069 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:15.369146 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:15.408478 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:15.983780 15106 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1016, Committed index: 1016, Last appended: 1.1016, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1017 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } }
I20240502 08:19:15.987133 16420 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1016. Preceding OpId from leader: term: 1 index: 1017. (index mismatch)
I20240502 08:19:15.994912 19514 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1017, Last known committed idx: 1016, Time since last communication: 0.000s
I20240502 08:19:16.001644 16420 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Committing config change with OpId 1.1017: config changed from index 129 to 1017, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 1017 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:16.006412 19202 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.1017: config changed from index 129 to 1017, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 1017 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:16.017658 14833 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: config changed from index 129 to 1017, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 1017 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } }
I20240502 08:19:16.110519 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2e9479648049ca94e7e5bfeb98db6e with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 1017) from {username='slave'} at 127.0.0.1:58056
I20240502 08:19:16.112306 19542 tablet_replica.cc:330] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:19:16.112578 19542 raft_consensus.cc:2229] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:16.112867 19542 raft_consensus.cc:2258] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:16.116096 19542 ts_tablet_manager.cc:1875] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:16.123311 19542 ts_tablet_manager.cc:1888] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1016
I20240502 08:19:16.123526 19542 log.cc:1196] T 6e2e9479648049ca94e7e5bfeb98db6e P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:16.124758 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet 6e2e9479648049ca94e7e5bfeb98db6e (table rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]) successfully deleted
I20240502 08:19:17.113047 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:17.143677 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:17.191486 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:17.231508 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:17.286051 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:17.305887 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:17.362008 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:19:17.446245 16484 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 115182us
W20240502 08:19:17.500397 14908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:44799, user_credentials={real_user=slave}} blocked reactor thread for 46458.7us
I20240502 08:19:17.945601 15106 consensus_queue.cc:237] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 952, Committed index: 952, Last appended: 1.952, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 953 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } }
I20240502 08:19:17.951190 16288 raft_consensus.cc:1261] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 952. Preceding OpId from leader: term: 1 index: 953. (index mismatch)
I20240502 08:19:17.952368 19513 consensus_queue.cc:1035] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 953, Last known committed idx: 952, Time since last communication: 0.000s
I20240502 08:19:17.960641 19627 raft_consensus.cc:2932] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.953: config changed from index 220 to 953, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 953 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:17.967384 16285 raft_consensus.cc:2932] T 73acd88beabf42679643daf4e62dea3b P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Committing config change with OpId 1.953: config changed from index 220 to 953, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 953 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:17.970830 14833 catalog_manager.cc:5484] T 73acd88beabf42679643daf4e62dea3b P 7fe0e31240d145a28e75262f706ebb89 reported cstate change: config changed from index 220 to 953, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 953 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } }
I20240502 08:19:18.122120 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet 73acd88beabf42679643daf4e62dea3b with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 953) from {username='slave'} at 127.0.0.1:55452
I20240502 08:19:18.136312 19646 tablet_replica.cc:330] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:19:18.138909 19646 raft_consensus.cc:2229] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:18.139290 19646 raft_consensus.cc:2258] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:18.155722 19646 ts_tablet_manager.cc:1875] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:18.173583 19646 ts_tablet_manager.cc:1888] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.952
I20240502 08:19:18.173907 19646 log.cc:1196] T 73acd88beabf42679643daf4e62dea3b P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/73acd88beabf42679643daf4e62dea3b
I20240502 08:19:18.175766 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet 73acd88beabf42679643daf4e62dea3b (table rebalance_test_table_3 [id=0d35af79bf4d4880a24ae8969568e211]) successfully deleted
W20240502 08:19:18.895236 16357 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:44789, user_credentials={real_user=slave}} blocked reactor thread for 58511.3us
I20240502 08:19:20.273514 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:20.281109 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:20.309391 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:20.337462 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:20.348517 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:20.474961 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:20.491043 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:21.321847 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:21.348593 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:21.412065 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:21.443198 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:21.543066 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:21.626988 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:21.659454 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:24.079488 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:24.087584 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:24.361526 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:24.390301 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.429203 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:24.463331 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:24.511324 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:25.387769 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:25.406419 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:25.525777 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:25.543712 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:25.577975 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:25.628715 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:25.681916 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:26.266840 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:26.274704 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:26.307557 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:26.327590 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:26.350764 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:26.352128 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:26.368110 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
481469713cc84d94a634ea9317661c18 | 127.8.68.254:44229 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.68.212:45753 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
19d550ab72684bf8aa1382d9c348cd29 | 127.8.68.195:43799 | HEALTHY | <none> | 5 | 0
382fd0c6daaf4eddb1c57557e9b716d1 | 127.8.68.197:41747 | HEALTHY | <none> | 1 | 0
7757b11bdda3443199e83298ba5c8118 | 127.8.68.199:34535 | HEALTHY | <none> | 0 | 0
7fe0e31240d145a28e75262f706ebb89 | 127.8.68.194:44789 | HEALTHY | <none> | 3 | 0
83659c8dbbcd4142abdbcb243eeec8d4 | 127.8.68.196:44799 | HEALTHY | <none> | 1 | 0
97037d47bf1249d2b5932b6553b907d4 | 127.8.68.193:33895 | HEALTHY | <none> | 5 | 0
a8cf9c3ffc34440594482a6718b596e5 | 127.8.68.198:34645 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.8.68.193 | experimental | 127.8.68.193:33895
local_ip_for_outbound_sockets | 127.8.68.194 | experimental | 127.8.68.194:44789
local_ip_for_outbound_sockets | 127.8.68.195 | experimental | 127.8.68.195:43799
local_ip_for_outbound_sockets | 127.8.68.196 | experimental | 127.8.68.196:44799
local_ip_for_outbound_sockets | 127.8.68.197 | experimental | 127.8.68.197:41747
local_ip_for_outbound_sockets | 127.8.68.198 | experimental | 127.8.68.198:34645
local_ip_for_outbound_sockets | 127.8.68.199 | experimental | 127.8.68.199:34535
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.8.68.193:33895
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.8.68.194:44789
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.8.68.195:43799
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.8.68.196:44799
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.8.68.197:41747
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.8.68.198:34645
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.8.68.199:34535
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.68.212:45753 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:19:26.486820 8467 log_verifier.cc:126] Checking tablet 1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:19:26.875300 8467 log_verifier.cc:177] Verified matching terms for 1145 ops in tablet 1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:19:26.876668 8467 log_verifier.cc:126] Checking tablet 445454fd0e3d405d87fc7ef20929e862
I20240502 08:19:27.329424 8467 log_verifier.cc:177] Verified matching terms for 1266 ops in tablet 445454fd0e3d405d87fc7ef20929e862
I20240502 08:19:27.330979 8467 log_verifier.cc:126] Checking tablet 4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:27.727087 8467 log_verifier.cc:177] Verified matching terms for 1156 ops in tablet 4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:27.728451 8467 log_verifier.cc:126] Checking tablet 4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:19:28.111147 8467 log_verifier.cc:177] Verified matching terms for 1113 ops in tablet 4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:19:28.112524 8467 log_verifier.cc:126] Checking tablet 4cc254d1803141fbab91423631ab8855
I20240502 08:19:28.488519 8467 log_verifier.cc:177] Verified matching terms for 1095 ops in tablet 4cc254d1803141fbab91423631ab8855
I20240502 08:19:28.489744 8467 log_verifier.cc:126] Checking tablet 4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:19:28.869205 8467 log_verifier.cc:177] Verified matching terms for 1161 ops in tablet 4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:19:28.870537 8467 log_verifier.cc:126] Checking tablet 68b0f3cfa17644449993f63affcbe2be
I20240502 08:19:29.258527 8467 log_verifier.cc:177] Verified matching terms for 1116 ops in tablet 68b0f3cfa17644449993f63affcbe2be
I20240502 08:19:29.259896 8467 log_verifier.cc:126] Checking tablet 6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:29.663713 8467 log_verifier.cc:177] Verified matching terms for 1196 ops in tablet 6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:29.665082 8467 log_verifier.cc:126] Checking tablet 73acd88beabf42679643daf4e62dea3b
I20240502 08:19:30.061975 8467 log_verifier.cc:177] Verified matching terms for 1099 ops in tablet 73acd88beabf42679643daf4e62dea3b
I20240502 08:19:30.063812 8467 log_verifier.cc:126] Checking tablet 8e685b7cf44341c5b9e89360d8488338
I20240502 08:19:30.396999 8467 log_verifier.cc:177] Verified matching terms for 932 ops in tablet 8e685b7cf44341c5b9e89360d8488338
I20240502 08:19:30.398056 8467 log_verifier.cc:126] Checking tablet 9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:30.695509 8467 log_verifier.cc:177] Verified matching terms for 877 ops in tablet 9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:30.696527 8467 log_verifier.cc:126] Checking tablet b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:31.022120 8467 log_verifier.cc:177] Verified matching terms for 922 ops in tablet b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:31.023167 8467 log_verifier.cc:126] Checking tablet cadca3ac819042f0bee9f3251283a6f3
I20240502 08:19:31.384617 8467 log_verifier.cc:177] Verified matching terms for 989 ops in tablet cadca3ac819042f0bee9f3251283a6f3
I20240502 08:19:31.385785 8467 log_verifier.cc:126] Checking tablet df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:19:31.728781 8467 log_verifier.cc:177] Verified matching terms for 956 ops in tablet df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:19:31.729843 8467 log_verifier.cc:126] Checking tablet ed128f6b7a67452db6b11ef0d14937dc
I20240502 08:19:31.745772 8467 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ed128f6b7a67452db6b11ef0d14937dc
I20240502 08:19:31.745874 8467 log_verifier.cc:126] Checking tablet f9acb74154404bd8829cc19db214068a
I20240502 08:19:32.085000 8467 log_verifier.cc:177] Verified matching terms for 1008 ops in tablet f9acb74154404bd8829cc19db214068a
I20240502 08:19:32.094630 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:0
--local_ip_for_outbound_sockets=127.8.68.200
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--builtin_ntp_servers=127.8.68.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:19:32.494791 20061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:32.495098 20061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:32.495235 20061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:32.503468 20061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:32.503602 20061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:32.503721 20061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240502 08:19:32.513130 20061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44229
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:32.514554 20061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:32.516414 20061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:32.530601 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:32.530882 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:32.531613 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:32.532796 20061 server_base.cc:1034] running on GCE node
I20240502 08:19:32.533494 20061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:32.536832 20061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:32.538705 20061 hybrid_clock.cc:648] HybridClock initialized: now 1714637972538611 us; error 127 us; skew 500 ppm
I20240502 08:19:32.539449 20061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:32.541985 20061 webserver.cc:460] Webserver started at http://127.8.68.200:41855/ using document root <none> and password file <none>
I20240502 08:19:32.542829 20061 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:32.542963 20061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:32.543320 20061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:32.546214 20061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5a76d98808654b17857cb67c2c3cef71"
format_stamp: "Formatted at 2024-05-02 08:19:32 on dist-test-slave-2dhs"
I20240502 08:19:32.547318 20061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5a76d98808654b17857cb67c2c3cef71"
format_stamp: "Formatted at 2024-05-02 08:19:32 on dist-test-slave-2dhs"
I20240502 08:19:32.552825 20061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:19:32.556406 20077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:32.557823 20061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 08:19:32.558010 20061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5a76d98808654b17857cb67c2c3cef71"
format_stamp: "Formatted at 2024-05-02 08:19:32 on dist-test-slave-2dhs"
I20240502 08:19:32.558230 20061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:32.582360 20061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:32.583324 20061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:32.583654 20061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:32.584761 20061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:32.586452 20061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:32.586563 20061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:32.586686 20061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:32.586778 20061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:32.621392 20061 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:34555
I20240502 08:19:32.621471 20190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:34555 every 8 connection(s)
I20240502 08:19:32.622785 20061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:19:32.629630 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 20061
I20240502 08:19:32.629890 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:19:32.639360 20191 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44229
I20240502 08:19:32.639755 20191 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:32.640525 20191 heartbeater.cc:511] Master 127.8.68.254:44229 requested a full tablet report, sending...
I20240502 08:19:32.641971 14827 ts_manager.cc:194] Registered new tserver with Master: 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555)
I20240502 08:19:32.643108 14827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:46215
I20240502 08:19:33.165830 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:33.172524 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.196640 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:33.200337 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:19:33.265702 15172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:34535, user_credentials={real_user=slave}} blocked reactor thread for 78019.8us
I20240502 08:19:33.298607 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.321062 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:33.334591 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.348621 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:33.541610 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:33.550323 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.563999 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:33.568727 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:33.571588 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.572757 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:33.577075 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:33.591507 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:33.646138 20191 heartbeater.cc:503] Master 127.8.68.254:44229 was elected leader, sending a full tablet report...
I20240502 08:19:33.707520 14975 consensus_queue.cc:237] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 877, Committed index: 877, Last appended: 1.877, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:33.710121 16288 raft_consensus.cc:1261] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 877. Preceding OpId from leader: term: 1 index: 878. (index mismatch)
I20240502 08:19:33.710745 19309 consensus_queue.cc:1035] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 878, Last known committed idx: 877, Time since last communication: 0.000s
W20240502 08:19:33.717973 14911 consensus_peers.cc:485] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9738c0a921014b7d9b8c9a1be4cbb433. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:33.720726 20298 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.878: config changed from index 266 to 878, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.721463 16288 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Committing config change with OpId 1.878: config changed from index 266 to 878, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.725793 14834 catalog_manager.cc:5484] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index 266 to 878, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.761590 15105 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1196, Committed index: 1196, Last appended: 1.1196, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:33.764637 16420 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Refusing update from remote peer 7fe0e31240d145a28e75262f706ebb89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1196. Preceding OpId from leader: term: 1 index: 1197. (index mismatch)
I20240502 08:19:33.765336 20321 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1197, Last known committed idx: 1196, Time since last communication: 0.000s
I20240502 08:19:33.769476 20042 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Committing config change with OpId 1.1197: config changed from index 1017 to 1197, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.770071 16420 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Committing config change with OpId 1.1197: config changed from index 1017 to 1197, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
W20240502 08:19:33.772836 15042 consensus_peers.cc:485] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e2e9479648049ca94e7e5bfeb98db6e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:33.775323 14828 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: config changed from index 1017 to 1197, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "7fe0e31240d145a28e75262f706ebb89" committed_config { opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.807008 14975 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1156, Committed index: 1156, Last appended: 1.1156, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:33.809201 16552 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1156. Preceding OpId from leader: term: 1 index: 1157. (index mismatch)
W20240502 08:19:33.809460 14911 consensus_peers.cc:485] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4485a1686beb45d7ad56010d2a7c9fc9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:33.809813 20298 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1157, Last known committed idx: 1156, Time since last communication: 0.000s
I20240502 08:19:33.812718 19309 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.1157: config changed from index 938 to 1157, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.813565 16552 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Committing config change with OpId 1.1157: config changed from index 938 to 1157, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.818336 14834 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: config changed from index 938 to 1157, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.851284 15234 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 922, Committed index: 922, Last appended: 1.922, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:33.854269 16683 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Refusing update from remote peer 19d550ab72684bf8aa1382d9c348cd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 922. Preceding OpId from leader: term: 1 index: 923. (index mismatch)
I20240502 08:19:33.854878 19851 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 923, Last known committed idx: 922, Time since last communication: 0.000s
I20240502 08:19:33.857687 19851 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Committing config change with OpId 1.923: config changed from index 519 to 923, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.858767 16683 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Committing config change with OpId 1.923: config changed from index 519 to 923, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New config: { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
W20240502 08:19:33.860213 15172 consensus_peers.cc:485] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2f290f16fe044658ed9b6b9a0a71f08. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:33.863651 14828 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 reported cstate change: config changed from index 519 to 923, VOTER 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "19d550ab72684bf8aa1382d9c348cd29" committed_config { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:33.986763 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:33.988044 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.017416 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:34.021915 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.024339 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:34.044057 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:34.066994 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:34.066994 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.230163 20418 ts_tablet_manager.cc:902] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:19:34.285286 20419 ts_tablet_manager.cc:902] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Initiating tablet copy from peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789)
I20240502 08:19:34.296455 20426 ts_tablet_manager.cc:902] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Initiating tablet copy from peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
I20240502 08:19:34.303010 20418 tablet_copy_client.cc:308] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:19:34.318203 14995 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 4485a1686beb45d7ad56010d2a7c9fc9 from peer 5a76d98808654b17857cb67c2c3cef71 ({username='slave'} at 127.8.68.200:47811)
I20240502 08:19:34.318442 14995 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 4485a1686beb45d7ad56010d2a7c9fc9 from peer 5a76d98808654b17857cb67c2c3cef71 at {username='slave'} at 127.8.68.200:47811: session id = 5a76d98808654b17857cb67c2c3cef71-4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:34.322798 14995 tablet_copy_source_session.cc:205] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:34.348722 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:34.373677 20419 tablet_copy_client.cc:308] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:44789
I20240502 08:19:34.384351 15126 tablet_copy_service.cc:140] P 7fe0e31240d145a28e75262f706ebb89: Received BeginTabletCopySession request for tablet 6e2e9479648049ca94e7e5bfeb98db6e from peer 5a76d98808654b17857cb67c2c3cef71 ({username='slave'} at 127.8.68.200:44975)
I20240502 08:19:34.384542 15126 tablet_copy_service.cc:161] P 7fe0e31240d145a28e75262f706ebb89: Beginning new tablet copy session on tablet 6e2e9479648049ca94e7e5bfeb98db6e from peer 5a76d98808654b17857cb67c2c3cef71 at {username='slave'} at 127.8.68.200:44975: session id = 5a76d98808654b17857cb67c2c3cef71-6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:34.387416 15126 tablet_copy_source_session.cc:205] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:34.394493 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2e9479648049ca94e7e5bfeb98db6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:34.430980 20439 ts_tablet_manager.cc:902] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Initiating tablet copy from peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
I20240502 08:19:34.431468 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:34.433517 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.442050 20426 tablet_copy_client.cc:308] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:33895
I20240502 08:19:34.443305 14994 tablet_copy_service.cc:140] P 97037d47bf1249d2b5932b6553b907d4: Received BeginTabletCopySession request for tablet 9738c0a921014b7d9b8c9a1be4cbb433 from peer 5a76d98808654b17857cb67c2c3cef71 ({username='slave'} at 127.8.68.200:47811)
I20240502 08:19:34.443464 14994 tablet_copy_service.cc:161] P 97037d47bf1249d2b5932b6553b907d4: Beginning new tablet copy session on tablet 9738c0a921014b7d9b8c9a1be4cbb433 from peer 5a76d98808654b17857cb67c2c3cef71 at {username='slave'} at 127.8.68.200:47811: session id = 5a76d98808654b17857cb67c2c3cef71-9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:34.444290 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.446429 14994 tablet_copy_source_session.cc:205] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:34.448338 20426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9738c0a921014b7d9b8c9a1be4cbb433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:34.450718 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:34.463295 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:34.486275 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:34.486902 20426 tablet_copy_client.cc:783] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:34.487381 20426 tablet_copy_client.cc:647] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:34.489401 20419 tablet_copy_client.cc:783] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:34.490916 20419 tablet_copy_client.cc:647] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:34.494624 20426 tablet_copy_client.cc:523] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:34.502490 20418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4485a1686beb45d7ad56010d2a7c9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:34.503645 20419 tablet_copy_client.cc:523] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:34.500597 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:34.509771 20418 tablet_copy_client.cc:783] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:34.510032 20418 tablet_copy_client.cc:647] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:34.511746 20439 tablet_copy_client.cc:308] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:43799
I20240502 08:19:34.516037 20426 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap starting.
I20240502 08:19:34.523619 15256 tablet_copy_service.cc:140] P 19d550ab72684bf8aa1382d9c348cd29: Received BeginTabletCopySession request for tablet b2f290f16fe044658ed9b6b9a0a71f08 from peer 5a76d98808654b17857cb67c2c3cef71 ({username='slave'} at 127.8.68.200:49449)
I20240502 08:19:34.523773 15256 tablet_copy_service.cc:161] P 19d550ab72684bf8aa1382d9c348cd29: Beginning new tablet copy session on tablet b2f290f16fe044658ed9b6b9a0a71f08 from peer 5a76d98808654b17857cb67c2c3cef71 at {username='slave'} at 127.8.68.200:49449: session id = 5a76d98808654b17857cb67c2c3cef71-b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:34.526592 15256 tablet_copy_source_session.cc:205] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:34.527637 20419 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Bootstrap starting.
I20240502 08:19:34.528141 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f290f16fe044658ed9b6b9a0a71f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:34.538260 20418 tablet_copy_client.cc:523] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:34.542151 20418 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap starting.
I20240502 08:19:34.556619 20439 tablet_copy_client.cc:783] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:34.557219 20439 tablet_copy_client.cc:647] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:34.570560 20439 tablet_copy_client.cc:523] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:34.575223 20439 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap starting.
I20240502 08:19:34.858947 20418 log.cc:824] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:35.023001 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:35.128268 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:35.137483 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:35.146021 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:35.148195 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:35.158519 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:35.194916 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:35.206089 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:35.769565 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:35.803151 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:35.841552 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:35.928712 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:35.932282 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:35.955552 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:35.975155 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:35.978933 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:36.475837 20426 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap replayed 1/1 log segments. Stats: ops{read=878 overwritten=0 applied=878 ignored=0} inserts{seen=1561 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:36.476540 20426 tablet_bootstrap.cc:492] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap complete.
I20240502 08:19:36.477248 20426 ts_tablet_manager.cc:1367] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Time spent bootstrapping tablet: real 1.978s user 1.065s sys 0.120s
I20240502 08:19:36.493214 20426 raft_consensus.cc:348] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:36.496814 20426 raft_consensus.cc:729] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a76d98808654b17857cb67c2c3cef71, State: Initialized, Role: FOLLOWER
I20240502 08:19:36.497767 20426 consensus_queue.cc:260] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 878, Last appended: 1.878, Last appended by leader: 878, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 878 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "83659c8dbbcd4142abdbcb243eeec8d4" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44799 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:36.515116 20426 ts_tablet_manager.cc:1398] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71: Time spent starting tablet: real 0.038s user 0.006s sys 0.005s
I20240502 08:19:36.516121 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 5a76d98808654b17857cb67c2c3cef71-9738c0a921014b7d9b8c9a1be4cbb433 received from {username='slave'} at 127.8.68.200:47811
I20240502 08:19:36.516239 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 5a76d98808654b17857cb67c2c3cef71-9738c0a921014b7d9b8c9a1be4cbb433 on tablet 9738c0a921014b7d9b8c9a1be4cbb433 with peer 5a76d98808654b17857cb67c2c3cef71
W20240502 08:19:36.522970 15042 consensus_peers.cc:485] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:36.596297 15172 consensus_peers.cc:485] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:36.615499 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:36.647878 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:36.668628 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:36.711757 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:36.735491 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:36.767074 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:36.778618 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:19:36.814697 14911 consensus_peers.cc:485] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 -> Peer 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555): Couldn't send request to peer 5a76d98808654b17857cb67c2c3cef71. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:36.834378 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:36.854972 20145 raft_consensus.cc:1203] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.877->[1.878-1.878] Dedup: 1.878->[]
I20240502 08:19:36.938591 20439 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap replayed 1/1 log segments. Stats: ops{read=923 overwritten=0 applied=923 ignored=0} inserts{seen=1606 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:36.939273 20439 tablet_bootstrap.cc:492] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap complete.
I20240502 08:19:36.939814 20439 ts_tablet_manager.cc:1367] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Time spent bootstrapping tablet: real 2.365s user 1.115s sys 0.102s
I20240502 08:19:36.940969 20439 raft_consensus.cc:348] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:36.941294 20439 raft_consensus.cc:729] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a76d98808654b17857cb67c2c3cef71, State: Initialized, Role: FOLLOWER
I20240502 08:19:36.941599 20439 consensus_queue.cc:260] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:36.942442 20439 ts_tablet_manager.cc:1398] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:36.943289 15256 tablet_copy_service.cc:342] P 19d550ab72684bf8aa1382d9c348cd29: Request end of tablet copy session 5a76d98808654b17857cb67c2c3cef71-b2f290f16fe044658ed9b6b9a0a71f08 received from {username='slave'} at 127.8.68.200:49449
I20240502 08:19:36.943411 15256 tablet_copy_service.cc:433] P 19d550ab72684bf8aa1382d9c348cd29: ending tablet copy session 5a76d98808654b17857cb67c2c3cef71-b2f290f16fe044658ed9b6b9a0a71f08 on tablet b2f290f16fe044658ed9b6b9a0a71f08 with peer 5a76d98808654b17857cb67c2c3cef71
I20240502 08:19:36.973839 20145 raft_consensus.cc:1203] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.922->[1.923-1.923] Dedup: 1.923->[]
I20240502 08:19:37.028082 20418 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap replayed 1/1 log segments. Stats: ops{read=1157 overwritten=0 applied=1157 ignored=0} inserts{seen=1786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:37.028646 20418 tablet_bootstrap.cc:492] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Bootstrap complete.
I20240502 08:19:37.029206 20418 ts_tablet_manager.cc:1367] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Time spent bootstrapping tablet: real 2.487s user 1.298s sys 0.131s
I20240502 08:19:37.030009 20418 raft_consensus.cc:348] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:37.030201 20418 raft_consensus.cc:729] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a76d98808654b17857cb67c2c3cef71, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.030611 20418 consensus_queue.cc:260] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1157, Last appended: 1.1157, Last appended by leader: 1157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:37.031203 20418 ts_tablet_manager.cc:1398] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:37.032151 14995 tablet_copy_service.cc:342] P 97037d47bf1249d2b5932b6553b907d4: Request end of tablet copy session 5a76d98808654b17857cb67c2c3cef71-4485a1686beb45d7ad56010d2a7c9fc9 received from {username='slave'} at 127.8.68.200:47811
I20240502 08:19:37.032269 14995 tablet_copy_service.cc:433] P 97037d47bf1249d2b5932b6553b907d4: ending tablet copy session 5a76d98808654b17857cb67c2c3cef71-4485a1686beb45d7ad56010d2a7c9fc9 on tablet 4485a1686beb45d7ad56010d2a7c9fc9 with peer 5a76d98808654b17857cb67c2c3cef71
I20240502 08:19:37.105352 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:37.114874 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.144217 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.157629 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.185679 20419 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Bootstrap replayed 1/1 log segments. Stats: ops{read=1197 overwritten=0 applied=1197 ignored=0} inserts{seen=1880 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:37.186265 20419 tablet_bootstrap.cc:492] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Bootstrap complete.
I20240502 08:19:37.187283 20419 ts_tablet_manager.cc:1367] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Time spent bootstrapping tablet: real 2.660s user 1.379s sys 0.115s
I20240502 08:19:37.188210 20419 raft_consensus.cc:348] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:37.188521 20419 raft_consensus.cc:729] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a76d98808654b17857cb67c2c3cef71, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.188813 20419 consensus_queue.cc:260] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1197, Last appended: 1.1197, Last appended by leader: 1197, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:37.190080 20419 ts_tablet_manager.cc:1398] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:19:37.191601 15126 tablet_copy_service.cc:342] P 7fe0e31240d145a28e75262f706ebb89: Request end of tablet copy session 5a76d98808654b17857cb67c2c3cef71-6e2e9479648049ca94e7e5bfeb98db6e received from {username='slave'} at 127.8.68.200:44975
I20240502 08:19:37.191717 15126 tablet_copy_service.cc:433] P 7fe0e31240d145a28e75262f706ebb89: ending tablet copy session 5a76d98808654b17857cb67c2c3cef71-6e2e9479648049ca94e7e5bfeb98db6e on tablet 6e2e9479648049ca94e7e5bfeb98db6e with peer 5a76d98808654b17857cb67c2c3cef71
I20240502 08:19:37.199545 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.202332 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:37.203640 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.206931 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:37.241771 20145 raft_consensus.cc:1203] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1156->[1.1157-1.1157] Dedup: 1.1157->[]
I20240502 08:19:37.400933 20145 raft_consensus.cc:1203] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1196->[1.1197-1.1197] Dedup: 1.1197->[]
I20240502 08:19:37.404839 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:37.416321 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.426352 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.454818 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.459118 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:37.467787 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.469213 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:37.488776 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.884011 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:37.887629 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.899793 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.913580 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:37.914367 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:37.918283 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:37.945511 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:38.011895 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.091652 15234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08"
dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57034
I20240502 08:19:38.091871 15234 raft_consensus.cc:595] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:38.195672 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.202299 16664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:38.211501 15216 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:38.215178 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:38.232245 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:38.239753 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.265784 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:38.316334 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.375000 20746 raft_consensus.cc:979] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: : Instructing follower 7757b11bdda3443199e83298ba5c8118 to start an election
I20240502 08:19:38.375175 20746 raft_consensus.cc:1067] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Signalling peer 7757b11bdda3443199e83298ba5c8118 to start an election
I20240502 08:19:38.376102 16683 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08"
dest_uuid: "7757b11bdda3443199e83298ba5c8118"
from {username='slave'} at 127.8.68.195:46983
I20240502 08:19:38.376461 16683 raft_consensus.cc:482] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:38.376605 16683 raft_consensus.cc:3037] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.379566 16683 raft_consensus.cc:504] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.385350 16683 leader_election.cc:288] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [CANDIDATE]: Term 2 election: Requested vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799), 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555)
I20240502 08:19:38.394517 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08" candidate_uuid: "7757b11bdda3443199e83298ba5c8118" candidate_term: 2 candidate_status { last_received { term: 1 index: 923 } } ignore_live_leader: true dest_uuid: "19d550ab72684bf8aa1382d9c348cd29"
I20240502 08:19:38.394759 15234 raft_consensus.cc:3032] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:38.394820 15234 raft_consensus.cc:729] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19d550ab72684bf8aa1382d9c348cd29, State: Running, Role: LEADER
I20240502 08:19:38.395048 15234 consensus_queue.cc:260] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 923, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.395704 15234 raft_consensus.cc:3037] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.398785 15234 raft_consensus.cc:2454] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7757b11bdda3443199e83298ba5c8118 in term 2.
I20240502 08:19:38.399034 20145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f290f16fe044658ed9b6b9a0a71f08" candidate_uuid: "7757b11bdda3443199e83298ba5c8118" candidate_term: 2 candidate_status { last_received { term: 1 index: 923 } } ignore_live_leader: true dest_uuid: "5a76d98808654b17857cb67c2c3cef71"
I20240502 08:19:38.399441 20145 raft_consensus.cc:3037] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.402568 20145 raft_consensus.cc:2454] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7757b11bdda3443199e83298ba5c8118 in term 2.
I20240502 08:19:38.403313 16620 leader_election.cc:302] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a76d98808654b17857cb67c2c3cef71, 7757b11bdda3443199e83298ba5c8118; no voters:
I20240502 08:19:38.404834 20754 raft_consensus.cc:2781] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:38.406090 20754 raft_consensus.cc:686] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 2 LEADER]: Becoming Leader. State: Replica: 7757b11bdda3443199e83298ba5c8118, State: Running, Role: LEADER
I20240502 08:19:38.406596 20754 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 923, Committed index: 923, Last appended: 1.923, Last appended by leader: 923, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.410116 14829 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 reported cstate change: term changed from 1 to 2, leader changed from 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) to 7757b11bdda3443199e83298ba5c8118 (127.8.68.199). New cstate: current_term: 2 leader_uuid: "7757b11bdda3443199e83298ba5c8118" committed_config { opid_index: 923 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:38.511224 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:38.544082 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.561223 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.579627 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:38.580758 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:38.581159 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.581856 16532 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.608062 14955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:38.690097 14975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9"
dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39720
I20240502 08:19:38.690286 14975 raft_consensus.cc:595] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:38.758432 20821 raft_consensus.cc:979] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: : Instructing follower a8cf9c3ffc34440594482a6718b596e5 to start an election
I20240502 08:19:38.758634 20341 raft_consensus.cc:1067] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Signalling peer a8cf9c3ffc34440594482a6718b596e5 to start an election
I20240502 08:19:38.759624 16551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9"
dest_uuid: "a8cf9c3ffc34440594482a6718b596e5"
from {username='slave'} at 127.8.68.193:41693
I20240502 08:19:38.759860 16551 raft_consensus.cc:482] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:38.759974 16551 raft_consensus.cc:3037] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.763520 16551 raft_consensus.cc:504] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.766294 16551 leader_election.cc:288] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 election: Requested vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895), 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200:34555)
I20240502 08:19:38.774585 20145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1157 } } ignore_live_leader: true dest_uuid: "5a76d98808654b17857cb67c2c3cef71"
I20240502 08:19:38.774858 20145 raft_consensus.cc:3037] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.777359 20145 raft_consensus.cc:2454] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 2.
I20240502 08:19:38.778020 16487 leader_election.cc:302] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a76d98808654b17857cb67c2c3cef71, a8cf9c3ffc34440594482a6718b596e5; no voters:
I20240502 08:19:38.778432 20668 raft_consensus.cc:2781] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:38.778718 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4485a1686beb45d7ad56010d2a7c9fc9" candidate_uuid: "a8cf9c3ffc34440594482a6718b596e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1157 } } ignore_live_leader: true dest_uuid: "97037d47bf1249d2b5932b6553b907d4"
I20240502 08:19:38.778944 14975 raft_consensus.cc:3032] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:38.779009 14975 raft_consensus.cc:729] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97037d47bf1249d2b5932b6553b907d4, State: Running, Role: LEADER
I20240502 08:19:38.779302 14975 consensus_queue.cc:260] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1157, Committed index: 1157, Last appended: 1.1157, Last appended by leader: 1157, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.779788 14975 raft_consensus.cc:3037] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:38.783975 14975 raft_consensus.cc:2454] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8cf9c3ffc34440594482a6718b596e5 in term 2.
I20240502 08:19:38.787549 20668 raft_consensus.cc:686] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 2 LEADER]: Becoming Leader. State: Replica: a8cf9c3ffc34440594482a6718b596e5, State: Running, Role: LEADER
I20240502 08:19:38.787865 20668 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1157, Committed index: 1157, Last appended: 1.1157, Last appended by leader: 1157, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:38.791280 14834 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: term changed from 1 to 2, leader changed from 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) to a8cf9c3ffc34440594482a6718b596e5 (127.8.68.198). New cstate: current_term: 2 leader_uuid: "a8cf9c3ffc34440594482a6718b596e5" committed_config { opid_index: 1157 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:38.828347 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.829710 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.831111 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:38.835439 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:38.841380 20145 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Refusing update from remote peer 7757b11bdda3443199e83298ba5c8118: Log matching property violated. Preceding OpId in replica: term: 1 index: 923. Preceding OpId from leader: term: 2 index: 924. (index mismatch)
I20240502 08:19:38.842418 20754 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 924, Last known committed idx: 923, Time since last communication: 0.000s
I20240502 08:19:38.851270 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:38.852605 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:38.864562 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:38.868021 15234 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Refusing update from remote peer 7757b11bdda3443199e83298ba5c8118: Log matching property violated. Preceding OpId in replica: term: 1 index: 923. Preceding OpId from leader: term: 2 index: 924. (index mismatch)
I20240502 08:19:38.868741 20754 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 924, Last known committed idx: 923, Time since last communication: 0.000s
I20240502 08:19:38.889835 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.095494 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.107107 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.109716 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:39.120941 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.144259 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:39.146816 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:39.162217 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.179288 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.224583 20145 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Refusing update from remote peer a8cf9c3ffc34440594482a6718b596e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1157. Preceding OpId from leader: term: 2 index: 1158. (index mismatch)
I20240502 08:19:39.225576 20668 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1158, Last known committed idx: 1157, Time since last communication: 0.000s
I20240502 08:19:39.251226 14975 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Refusing update from remote peer a8cf9c3ffc34440594482a6718b596e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1157. Preceding OpId from leader: term: 2 index: 1158. (index mismatch)
I20240502 08:19:39.259596 20825 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1158, Last known committed idx: 1157, Time since last communication: 0.000s
I20240502 08:19:39.317955 14975 consensus_queue.cc:237] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 878, Committed index: 878, Last appended: 1.878, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 879 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:39.320009 20145 raft_consensus.cc:1261] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Refusing update from remote peer 97037d47bf1249d2b5932b6553b907d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 878. Preceding OpId from leader: term: 1 index: 879. (index mismatch)
I20240502 08:19:39.320528 20821 consensus_queue.cc:1035] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 879, Last known committed idx: 878, Time since last communication: 0.000s
I20240502 08:19:39.323637 20906 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 [term 1 LEADER]: Committing config change with OpId 1.879: config changed from index 878 to 879, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) evicted. New config: { opid_index: 879 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:39.324249 20145 raft_consensus.cc:2932] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Committing config change with OpId 1.879: config changed from index 878 to 879, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) evicted. New config: { opid_index: 879 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:39.329111 14827 catalog_manager.cc:5484] T 9738c0a921014b7d9b8c9a1be4cbb433 P 97037d47bf1249d2b5932b6553b907d4 reported cstate change: config changed from index 878 to 879, VOTER 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "97037d47bf1249d2b5932b6553b907d4" committed_config { opid_index: 879 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:39.341495 16268 tablet_service.cc:1514] Processing DeleteTablet for tablet 9738c0a921014b7d9b8c9a1be4cbb433 with delete_type TABLET_DATA_TOMBSTONED (TS 83659c8dbbcd4142abdbcb243eeec8d4 not found in new config with opid_index 879) from {username='slave'} at 127.0.0.1:38130
I20240502 08:19:39.342309 20916 tablet_replica.cc:330] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: stopping tablet replica
I20240502 08:19:39.342732 20916 raft_consensus.cc:2229] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:39.343068 20916 raft_consensus.cc:2258] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:39.345923 20916 ts_tablet_manager.cc:1875] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:39.350795 20916 ts_tablet_manager.cc:1888] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.878
I20240502 08:19:39.350929 20916 log.cc:1196] T 9738c0a921014b7d9b8c9a1be4cbb433 P 83659c8dbbcd4142abdbcb243eeec8d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:39.351897 14818 catalog_manager.cc:4830] TS 83659c8dbbcd4142abdbcb243eeec8d4 (127.8.68.196:44799): tablet 9738c0a921014b7d9b8c9a1be4cbb433 (table rebalance_test_table_1 [id=170e980f97e343db86e5a897e1ba9a83]) successfully deleted
I20240502 08:19:39.446935 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.452790 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.453966 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.475906 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:39.477933 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:39.494155 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.497524 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.511193 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:39.596902 15105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e"
dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60386
I20240502 08:19:39.597095 15105 raft_consensus.cc:595] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:39.696503 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.717618 20125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:39.722177 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.724347 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.728713 15085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:39.737632 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:39.748090 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:39.789460 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:39.815380 20042 raft_consensus.cc:979] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: : Instructing follower 5a76d98808654b17857cb67c2c3cef71 to start an election
I20240502 08:19:39.815552 20042 raft_consensus.cc:1067] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Signalling peer 5a76d98808654b17857cb67c2c3cef71 to start an election
I20240502 08:19:39.816632 20145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e"
dest_uuid: "5a76d98808654b17857cb67c2c3cef71"
from {username='slave'} at 127.8.68.194:50975
I20240502 08:19:39.816973 20145 raft_consensus.cc:482] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:39.817130 20145 raft_consensus.cc:3037] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:39.820478 20145 raft_consensus.cc:504] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:39.823033 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e" candidate_uuid: "5a76d98808654b17857cb67c2c3cef71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "7fe0e31240d145a28e75262f706ebb89"
I20240502 08:19:39.823288 15105 raft_consensus.cc:3032] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:39.823370 15105 raft_consensus.cc:729] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fe0e31240d145a28e75262f706ebb89, State: Running, Role: LEADER
I20240502 08:19:39.823632 15105 consensus_queue.cc:260] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1197, Committed index: 1197, Last appended: 1.1197, Last appended by leader: 1197, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:39.823972 15105 raft_consensus.cc:3037] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:39.825513 20145 leader_election.cc:288] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [CANDIDATE]: Term 2 election: Requested vote from peers 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789), 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747)
I20240502 08:19:39.826838 15105 raft_consensus.cc:2454] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a76d98808654b17857cb67c2c3cef71 in term 2.
I20240502 08:19:39.827687 20081 leader_election.cc:302] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a76d98808654b17857cb67c2c3cef71, 7fe0e31240d145a28e75262f706ebb89; no voters:
I20240502 08:19:39.828146 20839 raft_consensus.cc:2781] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:39.836160 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2e9479648049ca94e7e5bfeb98db6e" candidate_uuid: "5a76d98808654b17857cb67c2c3cef71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "382fd0c6daaf4eddb1c57557e9b716d1"
I20240502 08:19:39.836800 16420 raft_consensus.cc:3037] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:39.840371 16420 raft_consensus.cc:2454] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a76d98808654b17857cb67c2c3cef71 in term 2.
I20240502 08:19:39.843550 20839 raft_consensus.cc:686] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 2 LEADER]: Becoming Leader. State: Replica: 5a76d98808654b17857cb67c2c3cef71, State: Running, Role: LEADER
I20240502 08:19:39.844247 20839 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1197, Committed index: 1197, Last appended: 1.1197, Last appended by leader: 1197, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:39.848053 14827 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 reported cstate change: term changed from 1 to 2, leader changed from 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) to 5a76d98808654b17857cb67c2c3cef71 (127.8.68.200). New cstate: current_term: 2 leader_uuid: "5a76d98808654b17857cb67c2c3cef71" committed_config { opid_index: 1197 OBSOLETE_local: false peers { permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 } } peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.181068 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:40.195597 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.207088 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.207794 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.213341 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.236425 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.244299 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.259022 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:40.294167 15105 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Refusing update from remote peer 5a76d98808654b17857cb67c2c3cef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1197. Preceding OpId from leader: term: 2 index: 1198. (index mismatch)
I20240502 08:19:40.294934 20839 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe0e31240d145a28e75262f706ebb89" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1198, Last known committed idx: 1197, Time since last communication: 0.000s
I20240502 08:19:40.312489 16420 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Refusing update from remote peer 5a76d98808654b17857cb67c2c3cef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1197. Preceding OpId from leader: term: 2 index: 1198. (index mismatch)
I20240502 08:19:40.313365 20839 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1198, Last known committed idx: 1197, Time since last communication: 0.000s
I20240502 08:19:40.451575 16683 consensus_queue.cc:237] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 924, Committed index: 924, Last appended: 2.924, Last appended by leader: 923, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:40.453367 20145 raft_consensus.cc:1261] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Refusing update from remote peer 7757b11bdda3443199e83298ba5c8118: Log matching property violated. Preceding OpId in replica: term: 2 index: 924. Preceding OpId from leader: term: 2 index: 925. (index mismatch)
I20240502 08:19:40.453909 20754 consensus_queue.cc:1035] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 925, Last known committed idx: 924, Time since last communication: 0.000s
I20240502 08:19:40.456708 21070 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 [term 2 LEADER]: Committing config change with OpId 2.925: config changed from index 923 to 925, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.457293 20145 raft_consensus.cc:2932] T b2f290f16fe044658ed9b6b9a0a71f08 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Committing config change with OpId 2.925: config changed from index 923 to 925, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.461194 14834 catalog_manager.cc:5484] T b2f290f16fe044658ed9b6b9a0a71f08 P 7757b11bdda3443199e83298ba5c8118 reported cstate change: config changed from index 923 to 925, VOTER 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195) evicted. New cstate: current_term: 2 leader_uuid: "7757b11bdda3443199e83298ba5c8118" committed_config { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.466404 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f290f16fe044658ed9b6b9a0a71f08 with delete_type TABLET_DATA_TOMBSTONED (TS 19d550ab72684bf8aa1382d9c348cd29 not found in new config with opid_index 925) from {username='slave'} at 127.0.0.1:55452
I20240502 08:19:40.467069 21079 tablet_replica.cc:330] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: stopping tablet replica
I20240502 08:19:40.467314 21079 raft_consensus.cc:2229] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:40.467592 21079 raft_consensus.cc:2258] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:40.469411 21079 ts_tablet_manager.cc:1875] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:40.476136 21079 ts_tablet_manager.cc:1888] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.924
I20240502 08:19:40.476298 21079 log.cc:1196] T b2f290f16fe044658ed9b6b9a0a71f08 P 19d550ab72684bf8aa1382d9c348cd29: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:40.477097 14820 catalog_manager.cc:4830] TS 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): tablet b2f290f16fe044658ed9b6b9a0a71f08 (table rebalance_test_table_0 [id=f4c7e164dcda4ea1a90e063191d0d9f0]) successfully deleted
I20240502 08:19:40.579221 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:40.594128 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.594740 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.594998 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.595676 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.596361 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.621723 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:40.642673 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.739462 16551 consensus_queue.cc:237] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1158, Committed index: 1158, Last appended: 2.1158, Last appended by leader: 1157, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:40.741312 20145 raft_consensus.cc:1261] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Refusing update from remote peer a8cf9c3ffc34440594482a6718b596e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1158. Preceding OpId from leader: term: 2 index: 1159. (index mismatch)
I20240502 08:19:40.741858 20668 consensus_queue.cc:1035] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1159, Last known committed idx: 1158, Time since last communication: 0.000s
I20240502 08:19:40.744601 21130 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 [term 2 LEADER]: Committing config change with OpId 2.1159: config changed from index 1157 to 1159, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.745226 20145 raft_consensus.cc:2932] T 4485a1686beb45d7ad56010d2a7c9fc9 P 5a76d98808654b17857cb67c2c3cef71 [term 2 FOLLOWER]: Committing config change with OpId 2.1159: config changed from index 1157 to 1159, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.749662 14834 catalog_manager.cc:5484] T 4485a1686beb45d7ad56010d2a7c9fc9 P a8cf9c3ffc34440594482a6718b596e5 reported cstate change: config changed from index 1157 to 1159, VOTER 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "a8cf9c3ffc34440594482a6718b596e5" committed_config { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:40.753506 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 4485a1686beb45d7ad56010d2a7c9fc9 with delete_type TABLET_DATA_TOMBSTONED (TS 97037d47bf1249d2b5932b6553b907d4 not found in new config with opid_index 1159) from {username='slave'} at 127.0.0.1:58056
I20240502 08:19:40.754117 21131 tablet_replica.cc:330] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: stopping tablet replica
I20240502 08:19:40.754376 21131 raft_consensus.cc:2229] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:40.754542 21131 raft_consensus.cc:2258] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:40.756827 21131 ts_tablet_manager.cc:1875] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:40.762943 21131 ts_tablet_manager.cc:1888] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1158
I20240502 08:19:40.763080 21131 log.cc:1196] T 4485a1686beb45d7ad56010d2a7c9fc9 P 97037d47bf1249d2b5932b6553b907d4: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:40.763952 14821 catalog_manager.cc:4830] TS 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): tablet 4485a1686beb45d7ad56010d2a7c9fc9 (table rebalance_test_table_2 [id=822e4752c7c24432937843b5ac7e12db]) successfully deleted
I20240502 08:19:40.865065 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.867705 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:40.888765 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.889230 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.889297 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.892819 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:40.898833 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:40.919745 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.032533 20145 consensus_queue.cc:237] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1198, Committed index: 1198, Last appended: 2.1198, Last appended by leader: 1197, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1199 OBSOLETE_local: false peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:41.034312 16420 raft_consensus.cc:1261] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Refusing update from remote peer 5a76d98808654b17857cb67c2c3cef71: Log matching property violated. Preceding OpId in replica: term: 2 index: 1198. Preceding OpId from leader: term: 2 index: 1199. (index mismatch)
I20240502 08:19:41.034965 21052 consensus_queue.cc:1035] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1199, Last known committed idx: 1198, Time since last communication: 0.000s
I20240502 08:19:41.037160 20991 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 [term 2 LEADER]: Committing config change with OpId 2.1199: config changed from index 1197 to 1199, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 1199 OBSOLETE_local: false peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:41.037614 16420 raft_consensus.cc:2932] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 [term 2 FOLLOWER]: Committing config change with OpId 2.1199: config changed from index 1197 to 1199, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New config: { opid_index: 1199 OBSOLETE_local: false peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:41.041981 14827 catalog_manager.cc:5484] T 6e2e9479648049ca94e7e5bfeb98db6e P 382fd0c6daaf4eddb1c57557e9b716d1 reported cstate change: config changed from index 1197 to 1199, VOTER 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "5a76d98808654b17857cb67c2c3cef71" committed_config { opid_index: 1199 OBSOLETE_local: false peers { permanent_uuid: "382fd0c6daaf4eddb1c57557e9b716d1" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 41747 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } } }
I20240502 08:19:41.047432 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2e9479648049ca94e7e5bfeb98db6e with delete_type TABLET_DATA_TOMBSTONED (TS 7fe0e31240d145a28e75262f706ebb89 not found in new config with opid_index 1199) from {username='slave'} at 127.0.0.1:43560
I20240502 08:19:41.048071 21196 tablet_replica.cc:330] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: stopping tablet replica
I20240502 08:19:41.048274 21196 raft_consensus.cc:2229] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:41.048444 21196 raft_consensus.cc:2258] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:41.050158 21196 ts_tablet_manager.cc:1875] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:41.057284 21196 ts_tablet_manager.cc:1888] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1198
I20240502 08:19:41.057489 21196 log.cc:1196] T 6e2e9479648049ca94e7e5bfeb98db6e P 7fe0e31240d145a28e75262f706ebb89: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:41.058455 14821 catalog_manager.cc:4830] TS 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): tablet 6e2e9479648049ca94e7e5bfeb98db6e (table rebalance_test_table_4 [id=d4c00a101b004fedac5027b3f62314cf]) successfully deleted
I20240502 08:19:41.196309 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.197054 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.208845 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.226593 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.238157 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.246990 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.269198 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.299461 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.523133 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.535615 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.538318 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.540155 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.543903 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.551565 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.559687 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.561193 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.943710 15216 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.947577 16401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.960283 16664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:41.967278 20125 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:42.004652 15085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:42.027173 14955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:42.034866 16268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:42.108668 16532 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
481469713cc84d94a634ea9317661c18 | 127.8.68.254:44229 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.68.212:45753 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
19d550ab72684bf8aa1382d9c348cd29 | 127.8.68.195:43799 | HEALTHY | <none> | 4 | 0
382fd0c6daaf4eddb1c57557e9b716d1 | 127.8.68.197:41747 | HEALTHY | <none> | 1 | 0
5a76d98808654b17857cb67c2c3cef71 | 127.8.68.200:34555 | HEALTHY | <none> | 1 | 0
7757b11bdda3443199e83298ba5c8118 | 127.8.68.199:34535 | HEALTHY | <none> | 1 | 0
7fe0e31240d145a28e75262f706ebb89 | 127.8.68.194:44789 | HEALTHY | <none> | 2 | 0
83659c8dbbcd4142abdbcb243eeec8d4 | 127.8.68.196:44799 | HEALTHY | <none> | 1 | 0
97037d47bf1249d2b5932b6553b907d4 | 127.8.68.193:33895 | HEALTHY | <none> | 4 | 0
a8cf9c3ffc34440594482a6718b596e5 | 127.8.68.198:34645 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.8.68.193 | experimental | 127.8.68.193:33895
local_ip_for_outbound_sockets | 127.8.68.194 | experimental | 127.8.68.194:44789
local_ip_for_outbound_sockets | 127.8.68.195 | experimental | 127.8.68.195:43799
local_ip_for_outbound_sockets | 127.8.68.196 | experimental | 127.8.68.196:44799
local_ip_for_outbound_sockets | 127.8.68.197 | experimental | 127.8.68.197:41747
local_ip_for_outbound_sockets | 127.8.68.198 | experimental | 127.8.68.198:34645
local_ip_for_outbound_sockets | 127.8.68.199 | experimental | 127.8.68.199:34535
local_ip_for_outbound_sockets | 127.8.68.200 | experimental | 127.8.68.200:34555
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.8.68.193:33895
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.8.68.194:44789
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.8.68.195:43799
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.8.68.196:44799
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.8.68.197:41747
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.8.68.198:34645
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.8.68.199:34535
server_dump_info_path | /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.8.68.200:34555
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.68.212:45753 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:19:42.158864 8467 log_verifier.cc:126] Checking tablet 1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:19:42.551916 8467 log_verifier.cc:177] Verified matching terms for 1145 ops in tablet 1aa2b2b96e134484a50edaac02dcc2a3
I20240502 08:19:42.553189 8467 log_verifier.cc:126] Checking tablet 445454fd0e3d405d87fc7ef20929e862
I20240502 08:19:43.003273 8467 log_verifier.cc:177] Verified matching terms for 1266 ops in tablet 445454fd0e3d405d87fc7ef20929e862
I20240502 08:19:43.004688 8467 log_verifier.cc:126] Checking tablet 4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:43.372574 8467 log_verifier.cc:177] Verified matching terms for 1159 ops in tablet 4485a1686beb45d7ad56010d2a7c9fc9
I20240502 08:19:43.373955 8467 log_verifier.cc:126] Checking tablet 4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:19:43.730069 8467 log_verifier.cc:177] Verified matching terms for 1113 ops in tablet 4c94e28f8f2b47cba77ed459c8e731d0
I20240502 08:19:43.731401 8467 log_verifier.cc:126] Checking tablet 4cc254d1803141fbab91423631ab8855
I20240502 08:19:44.103170 8467 log_verifier.cc:177] Verified matching terms for 1095 ops in tablet 4cc254d1803141fbab91423631ab8855
I20240502 08:19:44.104449 8467 log_verifier.cc:126] Checking tablet 4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:19:44.476800 8467 log_verifier.cc:177] Verified matching terms for 1161 ops in tablet 4e7ca0fbedbe43e2a42fd849c233ded3
I20240502 08:19:44.478127 8467 log_verifier.cc:126] Checking tablet 68b0f3cfa17644449993f63affcbe2be
I20240502 08:19:44.844444 8467 log_verifier.cc:177] Verified matching terms for 1116 ops in tablet 68b0f3cfa17644449993f63affcbe2be
I20240502 08:19:44.846033 8467 log_verifier.cc:126] Checking tablet 6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:45.221227 8467 log_verifier.cc:177] Verified matching terms for 1199 ops in tablet 6e2e9479648049ca94e7e5bfeb98db6e
I20240502 08:19:45.222556 8467 log_verifier.cc:126] Checking tablet 73acd88beabf42679643daf4e62dea3b
I20240502 08:19:45.586863 8467 log_verifier.cc:177] Verified matching terms for 1099 ops in tablet 73acd88beabf42679643daf4e62dea3b
I20240502 08:19:45.588064 8467 log_verifier.cc:126] Checking tablet 8e685b7cf44341c5b9e89360d8488338
I20240502 08:19:45.862092 8467 log_verifier.cc:177] Verified matching terms for 932 ops in tablet 8e685b7cf44341c5b9e89360d8488338
I20240502 08:19:45.863108 8467 log_verifier.cc:126] Checking tablet 9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:46.178331 8467 log_verifier.cc:177] Verified matching terms for 879 ops in tablet 9738c0a921014b7d9b8c9a1be4cbb433
I20240502 08:19:46.179472 8467 log_verifier.cc:126] Checking tablet b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:46.467229 8467 log_verifier.cc:177] Verified matching terms for 925 ops in tablet b2f290f16fe044658ed9b6b9a0a71f08
I20240502 08:19:46.468274 8467 log_verifier.cc:126] Checking tablet cadca3ac819042f0bee9f3251283a6f3
I20240502 08:19:46.793311 8467 log_verifier.cc:177] Verified matching terms for 989 ops in tablet cadca3ac819042f0bee9f3251283a6f3
I20240502 08:19:46.794282 8467 log_verifier.cc:126] Checking tablet df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:19:47.083345 8467 log_verifier.cc:177] Verified matching terms for 956 ops in tablet df91c5f0a46c43d78f8f7aeef7281e14
I20240502 08:19:47.084432 8467 log_verifier.cc:126] Checking tablet ed128f6b7a67452db6b11ef0d14937dc
I20240502 08:19:47.094735 8467 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ed128f6b7a67452db6b11ef0d14937dc
I20240502 08:19:47.094817 8467 log_verifier.cc:126] Checking tablet f9acb74154404bd8829cc19db214068a
I20240502 08:19:47.450593 8467 log_verifier.cc:177] Verified matching terms for 1008 ops in tablet f9acb74154404bd8829cc19db214068a
I20240502 08:19:47.455005 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14893
I20240502 08:19:47.475605 15014 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:47.779913 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14893
W20240502 08:19:47.832998 16487 connection.cc:535] server connection from 127.8.68.193:41693 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:47.833607 16356 connection.cc:535] server connection from 127.8.68.193:47741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:47.833770 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15024
W20240502 08:19:47.833925 20080 connection.cc:535] server connection from 127.8.68.193:49751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:47.839515 15145 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:48.159102 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15024
W20240502 08:19:48.212949 16221 connection.cc:535] server connection from 127.8.68.194:45709 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:48.213430 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 15154
I20240502 08:19:48.219476 15276 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:19:48.231184 16223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.194:44789: connect: Connection refused (error 111)
W20240502 08:19:48.232537 16223 consensus_peers.cc:485] T 68b0f3cfa17644449993f63affcbe2be P 83659c8dbbcd4142abdbcb243eeec8d4 -> Peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): Couldn't send request to peer 7fe0e31240d145a28e75262f706ebb89. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:44789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:48.378187 16357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.194:44789: connect: Connection refused (error 111)
W20240502 08:19:48.379650 16357 consensus_peers.cc:485] T df91c5f0a46c43d78f8f7aeef7281e14 P 382fd0c6daaf4eddb1c57557e9b716d1 -> Peer 7fe0e31240d145a28e75262f706ebb89 (127.8.68.194:44789): Couldn't send request to peer 7fe0e31240d145a28e75262f706ebb89. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:44789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:48.515753 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 15154
W20240502 08:19:48.587589 16223 connection.cc:535] server connection from 127.8.68.195:40713 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:48.587810 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 16205
W20240502 08:19:48.587888 16619 connection.cc:535] server connection from 127.8.68.195:46983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:48.594053 16328 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:48.870659 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 16205
W20240502 08:19:48.923617 14821 connection.cc:535] server connection from 127.8.68.196:43651 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:48.923810 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 16339
I20240502 08:19:48.930124 16460 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:49.195866 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 16339
W20240502 08:19:49.245745 14820 connection.cc:535] server connection from 127.8.68.197:53847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:49.245983 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 16469
I20240502 08:19:49.251912 16594 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:49.281484 21357 raft_consensus.cc:482] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97037d47bf1249d2b5932b6553b907d4)
I20240502 08:19:49.281742 21357 raft_consensus.cc:504] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 879 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "5a76d98808654b17857cb67c2c3cef71" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 34555 } }
I20240502 08:19:49.282477 21357 leader_election.cc:288] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895)
W20240502 08:19:49.283697 20081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:33895: connect: Connection refused (error 111)
W20240502 08:19:49.285115 20081 leader_election.cc:334] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 97037d47bf1249d2b5932b6553b907d4 (127.8.68.193:33895): Network error: Client connection negotiation failed: client connection to 127.8.68.193:33895: connect: Connection refused (error 111)
I20240502 08:19:49.285251 20081 leader_election.cc:302] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5a76d98808654b17857cb67c2c3cef71; no voters: 97037d47bf1249d2b5932b6553b907d4
I20240502 08:19:49.285531 21357 raft_consensus.cc:2726] T 9738c0a921014b7d9b8c9a1be4cbb433 P 5a76d98808654b17857cb67c2c3cef71 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:49.474627 20078 consensus_peers.cc:485] T 6e2e9479648049ca94e7e5bfeb98db6e P 5a76d98808654b17857cb67c2c3cef71 -> Peer 382fd0c6daaf4eddb1c57557e9b716d1 (127.8.68.197:41747): Couldn't send request to peer 382fd0c6daaf4eddb1c57557e9b716d1. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:41747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:49.507074 21353 raft_consensus.cc:482] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97037d47bf1249d2b5932b6553b907d4)
I20240502 08:19:49.507283 21353 raft_consensus.cc:504] T 445454fd0e3d405d87fc7ef20929e862 P a8cf9c3ffc34440594482a6718b596e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 777 OBSOLETE_local: false peers { permanent_uuid: "97037d47bf1249d2b5932b6553b907d4" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 33895 } } peers { permanent_uuid: "a8cf9c3ffc34440594482a6718b596e5" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 34645 } }
I20240502 08:19:49.507706 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 16469
I20240502 08:19:49.534602 21331 raft_consensus.cc:482] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19d550ab72684bf8aa1382d9c348cd29)
I20240502 08:19:49.534816 21331 raft_consensus.cc:504] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:19:49.535514 21331 leader_election.cc:288] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799)
W20240502 08:19:49.536592 16619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.195:43799: connect: Connection refused (error 111)
W20240502 08:19:49.537992 16619 leader_election.cc:334] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19d550ab72684bf8aa1382d9c348cd29 (127.8.68.195:43799): Network error: Client connection negotiation failed: client connection to 127.8.68.195:43799: connect: Connection refused (error 111)
I20240502 08:19:49.538151 16619 leader_election.cc:302] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7757b11bdda3443199e83298ba5c8118; no voters: 19d550ab72684bf8aa1382d9c348cd29
I20240502 08:19:49.538458 21331 raft_consensus.cc:2726] T 4cc254d1803141fbab91423631ab8855 P 7757b11bdda3443199e83298ba5c8118 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:19:49.556737 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 16602
I20240502 08:19:49.562697 16724 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:49.817798 21331 raft_consensus.cc:482] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19d550ab72684bf8aa1382d9c348cd29)
I20240502 08:19:49.818004 21331 raft_consensus.cc:504] T 4e7ca0fbedbe43e2a42fd849c233ded3 P 7757b11bdda3443199e83298ba5c8118 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 904 OBSOLETE_local: false peers { permanent_uuid: "19d550ab72684bf8aa1382d9c348cd29" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 43799 } } peers { permanent_uuid: "7757b11bdda3443199e83298ba5c8118" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 34535 } }
I20240502 08:19:49.818334 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 16602
I20240502 08:19:49.869328 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 20061
I20240502 08:19:49.876754 20186 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:50.087008 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 20061
W20240502 08:19:50.120898 14818 connection.cc:535] server connection from 127.8.68.200:46215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:50.121229 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 14803
I20240502 08:19:50.122253 14864 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:50.364532 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 14803
2024-05-02T08:19:50Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (117719 ms)
[----------] 1 test from RebalanceParamTest (117719 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 08:19:50.487169 8467 test_util.cc:268] Using random seed: 1146578018
I20240502 08:19:50.488842 8467 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:19:50.488917 8467 ts_itest-base.cc:113] --------------
I20240502 08:19:50.488971 8467 ts_itest-base.cc:114] 8 tablet servers
I20240502 08:19:50.489032 8467 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:19:50.489096 8467 ts_itest-base.cc:116] --------------
2024-05-02T08:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:19:50Z Disabled control of system clock
I20240502 08:19:50.556453 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:44613
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:44613
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 08:19:50.912257 21380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:50.912603 21380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:50.912753 21380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:50.920635 21380 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:50.920781 21380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:19:50.920845 21380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:50.920931 21380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:19:50.920994 21380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:19:50.930729 21380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:44613
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:50.932165 21380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:50.934031 21380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:50.945662 21385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:50.946355 21386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:50.946988 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:50.947842 21380 server_base.cc:1034] running on GCE node
I20240502 08:19:50.948758 21380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:50.950332 21380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:50.951550 21380 hybrid_clock.cc:648] HybridClock initialized: now 1714637990951502 us; error 51 us; skew 500 ppm
I20240502 08:19:50.951925 21380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:50.954478 21380 webserver.cc:460] Webserver started at http://127.8.68.254:40267/ using document root <none> and password file <none>
I20240502 08:19:50.955332 21380 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:50.955461 21380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:50.955807 21380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:50.958652 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8d832776683f415c8e7f1a1bc071aed9"
format_stamp: "Formatted at 2024-05-02 08:19:50 on dist-test-slave-2dhs"
I20240502 08:19:50.959539 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8d832776683f415c8e7f1a1bc071aed9"
format_stamp: "Formatted at 2024-05-02 08:19:50 on dist-test-slave-2dhs"
I20240502 08:19:50.965232 21380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:19:50.969054 21394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:50.970705 21380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:19:50.970911 21380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8d832776683f415c8e7f1a1bc071aed9"
format_stamp: "Formatted at 2024-05-02 08:19:50 on dist-test-slave-2dhs"
I20240502 08:19:50.971130 21380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:50.999096 21380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:51.000065 21380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:51.000557 21380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:51.020474 21380 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:44613
I20240502 08:19:51.020498 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:44613 every 8 connection(s)
I20240502 08:19:51.021883 21380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:19:51.024839 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21380
I20240502 08:19:51.025040 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:19:51.026238 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:51.040413 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9: Bootstrap starting.
I20240502 08:19:51.043581 21446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:51.044728 21446 log.cc:824] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:51.047317 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9: No bootstrap required, opened a new log
I20240502 08:19:51.053620 21446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } }
I20240502 08:19:51.053993 21446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:51.054092 21446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d832776683f415c8e7f1a1bc071aed9, State: Initialized, Role: FOLLOWER
I20240502 08:19:51.054688 21446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } }
I20240502 08:19:51.054894 21446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:19:51.055018 21446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:19:51.055174 21446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:51.057338 21446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } }
I20240502 08:19:51.057857 21446 leader_election.cc:302] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d832776683f415c8e7f1a1bc071aed9; no voters:
I20240502 08:19:51.058658 21446 leader_election.cc:288] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:19:51.058813 21451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:51.059497 21451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [term 1 LEADER]: Becoming Leader. State: Replica: 8d832776683f415c8e7f1a1bc071aed9, State: Running, Role: LEADER
I20240502 08:19:51.060088 21451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } }
I20240502 08:19:51.060706 21446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:19:51.062964 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d832776683f415c8e7f1a1bc071aed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } } }
I20240502 08:19:51.063355 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:51.063230 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d832776683f415c8e7f1a1bc071aed9. Latest consensus state: current_term: 1 leader_uuid: "8d832776683f415c8e7f1a1bc071aed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d832776683f415c8e7f1a1bc071aed9" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 44613 } } }
I20240502 08:19:51.063482 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:51.066129 21458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:19:51.075369 21458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:19:51.085196 21458 catalog_manager.cc:1347] Generated new cluster ID: 74639bd66fec4c5eb04d71b29881071e
I20240502 08:19:51.085345 21458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:19:51.130889 21458 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:19:51.132288 21458 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:19:51.147774 21458 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8d832776683f415c8e7f1a1bc071aed9: Generated new TSK 0
I20240502 08:19:51.148797 21458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:19:51.164505 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:19:51.560364 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:51.560690 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:51.560853 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:51.569157 21470 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:51.569311 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:51.569478 21470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240502 08:19:51.579021 21470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:51.580431 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:51.582178 21470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:51.595438 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:51.596014 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:51.596606 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:51.597424 21470 server_base.cc:1034] running on GCE node
I20240502 08:19:51.598093 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:51.599153 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:51.600380 21470 hybrid_clock.cc:648] HybridClock initialized: now 1714637991600307 us; error 69 us; skew 500 ppm
I20240502 08:19:51.600741 21470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:51.603192 21470 webserver.cc:460] Webserver started at http://127.8.68.193:43321/ using document root <none> and password file <none>
I20240502 08:19:51.604007 21470 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:51.604136 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:51.604478 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:51.607167 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cfff665685a14d8796aece26addab59c"
format_stamp: "Formatted at 2024-05-02 08:19:51 on dist-test-slave-2dhs"
I20240502 08:19:51.608062 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cfff665685a14d8796aece26addab59c"
format_stamp: "Formatted at 2024-05-02 08:19:51 on dist-test-slave-2dhs"
I20240502 08:19:51.613617 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:19:51.617583 21484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:51.619112 21470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:19:51.619325 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cfff665685a14d8796aece26addab59c"
format_stamp: "Formatted at 2024-05-02 08:19:51 on dist-test-slave-2dhs"
I20240502 08:19:51.619555 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:51.647377 21470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:51.648298 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:51.648591 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:51.649602 21470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:51.651192 21470 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:51.651333 21470 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240502 08:19:51.651448 21470 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:51.651521 21470 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:51.685530 21470 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:46845
I20240502 08:19:51.685588 21596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:46845 every 8 connection(s)
I20240502 08:19:51.686930 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:19:51.690910 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21470
I20240502 08:19:51.691160 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:19:51.694816 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:51.702229 21597 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:51.702558 21597 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:51.703435 21597 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:51.705613 21411 ts_manager.cc:194] Registered new tserver with Master: cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:19:51.707195 21411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:55929
W20240502 08:19:52.058528 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:52.058862 21601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:52.058961 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:52.066826 21601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:52.066962 21601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:52.067082 21601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240502 08:19:52.076506 21601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:52.078023 21601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:52.079818 21601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:52.092406 21609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:52.093091 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:52.093739 21606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:52.094383 21601 server_base.cc:1034] running on GCE node
I20240502 08:19:52.095000 21601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:52.096055 21601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:52.097276 21601 hybrid_clock.cc:648] HybridClock initialized: now 1714637992097222 us; error 49 us; skew 500 ppm
I20240502 08:19:52.097620 21601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:52.100057 21601 webserver.cc:460] Webserver started at http://127.8.68.194:40599/ using document root <none> and password file <none>
I20240502 08:19:52.100829 21601 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:52.100934 21601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:52.101258 21601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:52.103991 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "60677121510e47a5b51d1eb22f9aa238"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.104836 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "60677121510e47a5b51d1eb22f9aa238"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.110348 21601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 08:19:52.114033 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.115660 21601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:19:52.115846 21601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "60677121510e47a5b51d1eb22f9aa238"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.116060 21601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:52.132622 21601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:52.133563 21601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:52.133874 21601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:52.134989 21601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:52.136646 21601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:52.136755 21601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.136874 21601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:52.136947 21601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.171737 21601 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:36745
I20240502 08:19:52.171797 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:36745 every 8 connection(s)
I20240502 08:19:52.173082 21601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:19:52.175673 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21601
I20240502 08:19:52.175885 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:19:52.179556 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:52.194553 21728 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:52.195030 21728 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:52.196187 21728 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:52.197840 21411 ts_manager.cc:194] Registered new tserver with Master: 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:19:52.199059 21411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:59457
W20240502 08:19:52.555233 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:52.555675 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:52.555806 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:52.563841 21731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:52.563977 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:52.564168 21731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240502 08:19:52.573825 21731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:52.575294 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:52.577311 21731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:52.592411 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:52.592792 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:52.593375 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:52.594151 21731 server_base.cc:1034] running on GCE node
I20240502 08:19:52.594820 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:52.595980 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:52.597237 21731 hybrid_clock.cc:648] HybridClock initialized: now 1714637992597163 us; error 82 us; skew 500 ppm
I20240502 08:19:52.597630 21731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:52.611807 21731 webserver.cc:460] Webserver started at http://127.8.68.195:34019/ using document root <none> and password file <none>
I20240502 08:19:52.612895 21731 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:52.613072 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:52.613457 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:52.616470 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "13e764b27ed449c09aee719cdb2597f8"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.617332 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "13e764b27ed449c09aee719cdb2597f8"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.623307 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 08:19:52.627090 21745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.628608 21731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:19:52.628836 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "13e764b27ed449c09aee719cdb2597f8"
format_stamp: "Formatted at 2024-05-02 08:19:52 on dist-test-slave-2dhs"
I20240502 08:19:52.629047 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:52.647464 21731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:52.648389 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:52.648690 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:52.649709 21731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:52.651330 21731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:52.651434 21731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.651552 21731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:52.651639 21731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:52.685609 21731 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:33851
I20240502 08:19:52.685705 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:33851 every 8 connection(s)
I20240502 08:19:52.687346 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:19:52.687966 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21731
I20240502 08:19:52.688184 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:19:52.692633 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:52.710180 21597 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:19:52.716032 21858 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:52.716362 21858 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:52.717139 21858 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:52.718583 21411 ts_manager.cc:194] Registered new tserver with Master: 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:19:52.719450 21411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:48173
W20240502 08:19:53.065775 21862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:53.066123 21862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:53.066268 21862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:53.074404 21862 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:53.074548 21862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:53.074676 21862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240502 08:19:53.084496 21862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:53.085968 21862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:53.087759 21862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:53.100760 21867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:53.101454 21868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:53.102218 21870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:53.103020 21862 server_base.cc:1034] running on GCE node
I20240502 08:19:53.103753 21862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:53.104790 21862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:53.106019 21862 hybrid_clock.cc:648] HybridClock initialized: now 1714637993105958 us; error 44 us; skew 500 ppm
I20240502 08:19:53.106379 21862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:53.108925 21862 webserver.cc:460] Webserver started at http://127.8.68.196:35697/ using document root <none> and password file <none>
I20240502 08:19:53.109760 21862 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:53.109869 21862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:53.110219 21862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:53.113032 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0f655ab8f20d4ba482c3173f9594b98b"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.113962 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0f655ab8f20d4ba482c3173f9594b98b"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.119964 21862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 08:19:53.124054 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.125640 21862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:19:53.125847 21862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0f655ab8f20d4ba482c3173f9594b98b"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.126063 21862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:53.155732 21862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:53.156658 21862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:53.156948 21862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:53.158002 21862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:53.159693 21862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:53.159799 21862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.159919 21862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:53.159994 21862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.193796 21862 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:46395
I20240502 08:19:53.193877 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:46395 every 8 connection(s)
I20240502 08:19:53.195159 21862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:53.198688 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21862
I20240502 08:19:53.198879 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:19:53.201753 21728 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:19:53.202705 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:53.209542 21989 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:53.209862 21989 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:53.210630 21989 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:53.212131 21411 ts_manager.cc:194] Registered new tserver with Master: 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:19:53.213294 21411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:39295
W20240502 08:19:53.554582 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:53.554921 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:53.555047 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:53.562872 21992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:53.563019 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:53.563182 21992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:19:53.572597 21992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:53.573976 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:53.575847 21992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:53.589455 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:53.589957 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:53.590548 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:53.591395 21992 server_base.cc:1034] running on GCE node
I20240502 08:19:53.592070 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:53.593127 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:53.594357 21992 hybrid_clock.cc:648] HybridClock initialized: now 1714637993594315 us; error 42 us; skew 500 ppm
I20240502 08:19:53.594691 21992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:53.597213 21992 webserver.cc:460] Webserver started at http://127.8.68.197:40675/ using document root <none> and password file <none>
I20240502 08:19:53.598004 21992 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:53.598141 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:53.598480 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:53.601387 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "20b637fbec5444dd91b3af9cdda482f5"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.602221 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "20b637fbec5444dd91b3af9cdda482f5"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.607935 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:19:53.612372 22006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.614068 21992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:19:53.614286 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "20b637fbec5444dd91b3af9cdda482f5"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:19:53.614524 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:53.636647 21992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:53.637559 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:53.637881 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:53.638934 21992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:53.640576 21992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:53.640689 21992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.640810 21992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:53.640883 21992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:53.674639 21992 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:45373
I20240502 08:19:53.674693 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:45373 every 8 connection(s)
I20240502 08:19:53.676144 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:19:53.682080 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 21992
I20240502 08:19:53.682313 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:19:53.685667 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:53.695343 22119 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:53.695740 22119 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:53.696585 22119 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:53.698083 21411 ts_manager.cc:194] Registered new tserver with Master: 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:19:53.698966 21411 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:51835
I20240502 08:19:53.722095 21858 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
W20240502 08:19:54.052932 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:54.053251 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:54.053395 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:54.061149 22122 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:54.061300 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:54.061466 22122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:19:54.070803 22122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:54.072218 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:54.073995 22122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:54.086745 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:54.087216 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:54.087946 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:54.088723 22122 server_base.cc:1034] running on GCE node
I20240502 08:19:54.089355 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:54.090338 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:54.091545 22122 hybrid_clock.cc:648] HybridClock initialized: now 1714637994091465 us; error 71 us; skew 500 ppm
I20240502 08:19:54.091879 22122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:54.094378 22122 webserver.cc:460] Webserver started at http://127.8.68.198:39265/ using document root <none> and password file <none>
I20240502 08:19:54.095149 22122 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:54.095304 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:54.095638 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:54.098315 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4b9bfd2541144dad86507451343640d3"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.099149 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4b9bfd2541144dad86507451343640d3"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.104780 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 08:19:54.108493 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.109932 22122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:19:54.110123 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b9bfd2541144dad86507451343640d3"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.110344 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:54.139122 22122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:54.140084 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:54.140383 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:54.141440 22122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:54.143059 22122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:54.143162 22122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.143311 22122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:54.143388 22122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.177901 22122 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:41299
I20240502 08:19:54.177938 22248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:41299 every 8 connection(s)
I20240502 08:19:54.179302 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:19:54.189672 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 22122
I20240502 08:19:54.189889 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:19:54.192873 22249 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:54.193276 22249 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:54.193663 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:0
--local_ip_for_outbound_sockets=127.8.68.199
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:54.194245 22249 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:54.195991 21410 ts_manager.cc:194] Registered new tserver with Master: 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299)
I20240502 08:19:54.197481 21410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:42235
I20240502 08:19:54.215955 21989 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
W20240502 08:19:54.551553 22252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:54.551908 22252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:54.552067 22252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:54.560953 22252 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:54.561120 22252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:54.561295 22252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:19:54.570858 22252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:54.572391 22252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:54.574283 22252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:54.587677 22257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:54.588194 22258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:54.588909 22260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:54.589641 22252 server_base.cc:1034] running on GCE node
I20240502 08:19:54.590322 22252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:54.591445 22252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:54.592681 22252 hybrid_clock.cc:648] HybridClock initialized: now 1714637994592627 us; error 48 us; skew 500 ppm
I20240502 08:19:54.593058 22252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:54.595786 22252 webserver.cc:460] Webserver started at http://127.8.68.199:36495/ using document root <none> and password file <none>
I20240502 08:19:54.596642 22252 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:54.596756 22252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:54.597134 22252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:54.603682 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.604619 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.610705 22252 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 08:19:54.614876 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.616632 22252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:19:54.616837 22252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:19:54.617074 22252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:54.632055 22252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:54.632972 22252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:54.633325 22252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:54.634470 22252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:54.636269 22252 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:54.636376 22252 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.636503 22252 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:54.636577 22252 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:54.669981 22252 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:33397
I20240502 08:19:54.670082 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:33397 every 8 connection(s)
I20240502 08:19:54.671442 22252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:19:54.674255 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 22252
I20240502 08:19:54.674472 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:19:54.678943 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:0
--local_ip_for_outbound_sockets=127.8.68.200
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:19:54.692816 22379 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:54.693230 22379 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:54.694353 22379 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:54.696369 21410 ts_manager.cc:194] Registered new tserver with Master: fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397)
I20240502 08:19:54.697259 21410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:38793
I20240502 08:19:54.701671 22119 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
W20240502 08:19:55.049932 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:55.050261 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:55.050387 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:55.058014 22382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:19:55.058159 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:55.058279 22382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240502 08:19:55.067917 22382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:55.069245 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:55.071036 22382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:55.083388 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:55.083771 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:55.084308 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:55.085055 22382 server_base.cc:1034] running on GCE node
I20240502 08:19:55.085669 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:55.086659 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:55.087908 22382 hybrid_clock.cc:648] HybridClock initialized: now 1714637995087859 us; error 41 us; skew 500 ppm
I20240502 08:19:55.088258 22382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:55.090787 22382 webserver.cc:460] Webserver started at http://127.8.68.200:44633/ using document root <none> and password file <none>
I20240502 08:19:55.091674 22382 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:55.091799 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:55.092132 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:55.094885 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "95f32202b9db457fb09872f6f51ea8ba"
format_stamp: "Formatted at 2024-05-02 08:19:55 on dist-test-slave-2dhs"
I20240502 08:19:55.095733 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "95f32202b9db457fb09872f6f51ea8ba"
format_stamp: "Formatted at 2024-05-02 08:19:55 on dist-test-slave-2dhs"
I20240502 08:19:55.101260 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:19:55.105023 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:55.106560 22382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:19:55.106775 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "95f32202b9db457fb09872f6f51ea8ba"
format_stamp: "Formatted at 2024-05-02 08:19:55 on dist-test-slave-2dhs"
I20240502 08:19:55.106981 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:55.125530 22382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:19:55.126458 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:55.126750 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:55.127869 22382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:55.129478 22382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:55.129582 22382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:55.129671 22382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:55.129747 22382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:19:55.163973 22382 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:40881
I20240502 08:19:55.164058 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:40881 every 8 connection(s)
I20240502 08:19:55.165330 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:19:55.173024 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 22382
I20240502 08:19:55.173261 8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:19:55.179050 22509 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:19:55.179447 22509 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:55.180444 22509 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:19:55.182077 21410 ts_manager.cc:194] Registered new tserver with Master: 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
I20240502 08:19:55.182891 21410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:49813
I20240502 08:19:55.190171 8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:19:55.200567 22249 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:19:55.208515 8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:19:55.208659 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 21992
I20240502 08:19:55.217015 22114 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.301620 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 21992
I20240502 08:19:55.318645 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 22122
I20240502 08:19:55.327572 22244 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.410663 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 22122
I20240502 08:19:55.427929 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 22252
I20240502 08:19:55.436514 22374 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.519193 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 22252
I20240502 08:19:55.538637 8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskivhcCF/build/asan/bin/kudu pid 22382
I20240502 08:19:55.547140 22504 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.628413 8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskivhcCF/build/asan/bin/kudu with pid 22382
I20240502 08:19:59.396934 21411 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:19:59.450428 21792 tablet_service.cc:1467] Processing CreateTablet for tablet ebf79c69fd614284ba45fd052cac7ebf (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.451949 21793 tablet_service.cc:1467] Processing CreateTablet for tablet d4d933eacd944d84b39f0cb9c45641f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.453248 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.454632 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf79c69fd614284ba45fd052cac7ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.457690 21662 tablet_service.cc:1467] Processing CreateTablet for tablet ebf79c69fd614284ba45fd052cac7ebf (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.457701 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.459396 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.459779 21663 tablet_service.cc:1467] Processing CreateTablet for tablet d4d933eacd944d84b39f0cb9c45641f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.460556 21661 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.460878 21924 tablet_service.cc:1467] Processing CreateTablet for tablet d4d933eacd944d84b39f0cb9c45641f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.461426 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.465359 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf79c69fd614284ba45fd052cac7ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.466239 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.467846 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.469347 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.470306 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.467025 21532 tablet_service.cc:1467] Processing CreateTablet for tablet ebf79c69fd614284ba45fd052cac7ebf (DEFAULT_TABLE table=rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:59.473389 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf79c69fd614284ba45fd052cac7ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:59.495288 22539 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:19:59.496598 22540 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:19:59.499637 22539 tablet_bootstrap.cc:654] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.500463 22540 tablet_bootstrap.cc:654] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.501156 22539 log.cc:824] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:59.501905 22540 log.cc:824] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:59.512450 22539 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:19:59.512943 22539 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20240502 08:19:59.516808 22538 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:19:59.517958 22543 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:19:59.518429 22540 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:19:59.518821 22540 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.012s
I20240502 08:19:59.521032 22538 tablet_bootstrap.cc:654] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.522733 22543 tablet_bootstrap.cc:654] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.522689 22538 log.cc:824] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:59.524466 22543 log.cc:824] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:59.529148 22539 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.530184 22539 raft_consensus.cc:374] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.530436 22539 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.531492 22540 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.531739 22539 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.532169 22540 raft_consensus.cc:374] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.532291 22540 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.533536 22540 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.534956 22539 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.022s user 0.017s sys 0.000s
I20240502 08:19:59.535559 22539 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:19:59.535940 22543 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:19:59.536301 22543 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20240502 08:19:59.536655 22538 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:19:59.536952 22538 ts_tablet_manager.cc:1367] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.013s
I20240502 08:19:59.537720 22539 tablet_bootstrap.cc:654] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.543982 22540 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.025s user 0.016s sys 0.002s
I20240502 08:19:59.544492 22539 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:19:59.544766 22540 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:19:59.544785 22539 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 08:19:59.545950 22539 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.546208 22539 raft_consensus.cc:374] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.546304 22539 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.546582 22539 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.547024 22540 tablet_bootstrap.cc:654] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.548420 22539 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:19:59.548272 22538 raft_consensus.cc:348] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.548950 22538 raft_consensus.cc:374] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.549083 22538 raft_consensus.cc:729] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.549453 22547 raft_consensus.cc:482] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:59.549773 22547 raft_consensus.cc:504] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.550036 22538 consensus_queue.cc:260] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.551643 22540 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:19:59.551846 22540 ts_tablet_manager.cc:1367] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:19:59.552414 22547 leader_election.cc:288] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:19:59.553062 22540 raft_consensus.cc:348] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.553290 22540 raft_consensus.cc:374] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.553380 22540 raft_consensus.cc:729] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.553666 22540 consensus_queue.cc:260] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.554260 22540 ts_tablet_manager.cc:1398] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:59.556584 22538 ts_tablet_manager.cc:1398] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.020s user 0.008s sys 0.008s
I20240502 08:19:59.557307 22543 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.557926 22543 raft_consensus.cc:374] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.558102 22543 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.559005 22538 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:19:59.559036 22543 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.562568 22538 tablet_bootstrap.cc:654] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.575407 22543 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.039s user 0.014s sys 0.015s
I20240502 08:19:59.576054 22543 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:19:59.576570 22538 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:19:59.576452 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:19:59.576815 22538 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240502 08:19:59.577070 21944 raft_consensus.cc:2454] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:19:59.578220 21488 leader_election.cc:302] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:19:59.578397 22543 tablet_bootstrap.cc:654] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.578241 22538 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.578477 22538 raft_consensus.cc:374] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.578634 22538 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.578868 22547 raft_consensus.cc:2781] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:59.578892 22538 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.579061 22547 raft_consensus.cc:482] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:59.579187 22547 raft_consensus.cc:3037] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.579982 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:19:59.582207 22538 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:19:59.582983 22547 raft_consensus.cc:504] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
W20240502 08:19:59.583137 21488 leader_election.cc:341] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:19:59.583839 22547 leader_election.cc:288] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:19:59.584491 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:19:59.584597 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:19:59.584755 21944 raft_consensus.cc:3037] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:19:59.585196 21488 leader_election.cc:341] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:19:59.585270 22543 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:19:59.585496 22543 ts_tablet_manager.cc:1367] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 08:19:59.586534 22543 raft_consensus.cc:348] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.586717 22543 raft_consensus.cc:374] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.586820 22543 raft_consensus.cc:729] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.587082 22543 consensus_queue.cc:260] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.587615 22543 ts_tablet_manager.cc:1398] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:59.587811 21944 raft_consensus.cc:2454] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:19:59.587901 22543 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:19:59.588443 21488 leader_election.cc:302] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c; no voters: 60677121510e47a5b51d1eb22f9aa238
I20240502 08:19:59.588948 22547 raft_consensus.cc:2781] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:59.589782 22543 tablet_bootstrap.cc:654] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:59.590109 22547 raft_consensus.cc:686] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Becoming Leader. State: Replica: cfff665685a14d8796aece26addab59c, State: Running, Role: LEADER
I20240502 08:19:59.590727 22547 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.592278 22543 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:19:59.592530 22543 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240502 08:19:59.593184 22543 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.593400 22543 raft_consensus.cc:374] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:59.593477 22543 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:19:59.593744 22543 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:19:59.594276 22543 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:19:59.595934 21410 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c reported cstate change: term changed from 0 to 1, leader changed from <none> to cfff665685a14d8796aece26addab59c (127.8.68.193). New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
W20240502 08:19:59.680920 21729 tablet.cc:2369] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:19:59.696092 21598 tablet.cc:2369] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:19:59.702071 21990 tablet.cc:2369] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:19:59.714051 21859 tablet.cc:2369] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:19:59.758344 22553 raft_consensus.cc:482] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:59.758769 22553 raft_consensus.cc:504] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.760592 22553 leader_election.cc:288] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:19:59.770385 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d933eacd944d84b39f0cb9c45641f8" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:19:59.770799 21944 raft_consensus.cc:2454] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:19:59.770766 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d933eacd944d84b39f0cb9c45641f8" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:19:59.771452 21813 raft_consensus.cc:2454] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:19:59.771495 21619 leader_election.cc:302] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:19:59.772019 22553 raft_consensus.cc:2781] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:59.772168 22553 raft_consensus.cc:482] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:59.772277 22553 raft_consensus.cc:3037] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.774521 22553 raft_consensus.cc:504] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.775288 22553 leader_election.cc:288] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:19:59.776038 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d933eacd944d84b39f0cb9c45641f8" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:19:59.776064 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d933eacd944d84b39f0cb9c45641f8" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:19:59.776324 21944 raft_consensus.cc:3037] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.776311 21813 raft_consensus.cc:3037] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.779601 21944 raft_consensus.cc:2454] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:19:59.779614 21813 raft_consensus.cc:2454] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:19:59.780267 21618 leader_election.cc:302] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:19:59.780648 22553 raft_consensus.cc:2781] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:59.781087 22553 raft_consensus.cc:686] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:19:59.781634 22553 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:19:59.786250 21410 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:19:59.827505 22550 raft_consensus.cc:482] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:59.827752 22550 raft_consensus.cc:504] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.829509 22550 leader_election.cc:288] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:19:59.837566 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf79c69fd614284ba45fd052cac7ebf" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:19:59.837868 21683 raft_consensus.cc:2454] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:19:59.838131 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf79c69fd614284ba45fd052cac7ebf" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:19:59.838361 21749 leader_election.cc:302] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:19:59.838594 21552 raft_consensus.cc:2454] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:19:59.838819 22550 raft_consensus.cc:2781] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:59.838953 22550 raft_consensus.cc:482] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:59.839073 22550 raft_consensus.cc:3037] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.841173 22550 raft_consensus.cc:504] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.841835 22550 leader_election.cc:288] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:19:59.842376 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf79c69fd614284ba45fd052cac7ebf" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:19:59.842449 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf79c69fd614284ba45fd052cac7ebf" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:19:59.842583 21683 raft_consensus.cc:3037] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.842725 21552 raft_consensus.cc:3037] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:59.844849 21683 raft_consensus.cc:2454] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:19:59.844907 21552 raft_consensus.cc:2454] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:19:59.845397 21749 leader_election.cc:302] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:19:59.845767 22550 raft_consensus.cc:2781] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:59.846235 22550 raft_consensus.cc:686] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:19:59.846720 22550 consensus_queue.cc:237] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:19:59.851723 21410 catalog_manager.cc:5484] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:20:00.058823 22553 raft_consensus.cc:482] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:00.059103 22553 raft_consensus.cc:504] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:00.059938 22553 leader_election.cc:288] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:00.060590 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:00.060914 21944 raft_consensus.cc:2379] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60677121510e47a5b51d1eb22f9aa238 in current term 1: Already voted for candidate cfff665685a14d8796aece26addab59c in this term.
I20240502 08:20:00.066998 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0a886b3ba04737a6988c3b58c9698e" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:00.067795 21619 leader_election.cc:302] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238; no voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c
I20240502 08:20:00.068158 22553 raft_consensus.cc:3037] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:00.071019 22553 raft_consensus.cc:2726] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:20:00.072687 22547 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:00.083374 22556 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:00.287083 22567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:00.287516 22567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:00.295739 22567 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:20:00.295926 22553 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:00.302364 22578 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 08:20:00.370285 22550 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:00.409385 22550 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 300.415 ms
time per row: 0.150207 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:20:01.732929 21403 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:01.743093 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 2de41d6c6de0400397645c05a0fa17de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.743330 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 2de41d6c6de0400397645c05a0fa17de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.743687 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.743789 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.744863 21532 tablet_service.cc:1467] Processing CreateTablet for tablet dc49c050280346db88d1323e30513c80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.745270 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.745756 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 3898bfdb9c3d4a05af78f79fd2de61e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.746189 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3898bfdb9c3d4a05af78f79fd2de61e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.747694 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 2de41d6c6de0400397645c05a0fa17de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.748098 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.751756 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 3898bfdb9c3d4a05af78f79fd2de61e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.752403 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3898bfdb9c3d4a05af78f79fd2de61e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.755894 21924 tablet_service.cc:1467] Processing CreateTablet for tablet dc49c050280346db88d1323e30513c80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.756556 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.758483 22636 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:01.760078 21793 tablet_service.cc:1467] Processing CreateTablet for tablet dc49c050280346db88d1323e30513c80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.760398 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 3898bfdb9c3d4a05af78f79fd2de61e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:01.760596 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.760654 22636 tablet_bootstrap.cc:654] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.760958 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3898bfdb9c3d4a05af78f79fd2de61e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:01.776087 22637 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:01.779678 22637 tablet_bootstrap.cc:654] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.781226 22636 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:01.781523 22636 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20240502 08:20:01.782512 22636 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.784142 22636 raft_consensus.cc:374] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.784379 22636 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.784794 22636 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.785169 22639 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:01.786424 22636 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:20:01.786761 22636 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:01.787225 22639 tablet_bootstrap.cc:654] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.788843 22636 tablet_bootstrap.cc:654] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.789742 22635 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:01.790019 22637 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:01.790251 22637 ts_tablet_manager.cc:1367] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 08:20:01.791116 22637 raft_consensus.cc:348] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.791424 22637 raft_consensus.cc:374] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.791523 22637 raft_consensus.cc:729] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.791935 22637 consensus_queue.cc:260] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.792706 22637 ts_tablet_manager.cc:1398] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:20:01.793054 22639 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:01.793076 22635 tablet_bootstrap.cc:654] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.793234 22639 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 08:20:01.793702 22636 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:01.793927 22636 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:20:01.794095 22639 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.794325 22639 raft_consensus.cc:374] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.794449 22639 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.794726 22639 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.795158 22636 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.795614 22639 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:01.796015 22639 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:01.795778 22636 raft_consensus.cc:374] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.796401 22636 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.796661 22636 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.796828 22635 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:01.796969 22635 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 08:20:01.797232 22636 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:20:01.797766 22639 tablet_bootstrap.cc:654] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.797884 22635 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.798214 22636 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:01.798287 22635 raft_consensus.cc:374] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.798380 22635 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.798674 22635 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.800171 22636 tablet_bootstrap.cc:654] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.800201 22635 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:20:01.800499 22635 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:01.802274 22635 tablet_bootstrap.cc:654] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.809567 22635 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:01.809784 22635 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240502 08:20:01.810091 22636 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:01.810256 22636 ts_tablet_manager.cc:1367] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20240502 08:20:01.810644 22635 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.811095 22635 raft_consensus.cc:374] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.811184 22639 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:01.811049 22636 raft_consensus.cc:348] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.811178 22635 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.811298 22636 raft_consensus.cc:374] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.811393 22639 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20240502 08:20:01.811407 22636 raft_consensus.cc:729] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.811455 22635 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.811626 22636 consensus_queue.cc:260] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.812230 22636 ts_tablet_manager.cc:1398] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:01.812569 22635 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:20:01.812845 22639 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.813076 22639 raft_consensus.cc:374] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.813149 22639 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.813211 22635 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:01.813417 22639 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.813933 22639 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:20:01.815351 22635 tablet_bootstrap.cc:654] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:01.817646 22635 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:01.817822 22635 ts_tablet_manager.cc:1367] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:20:01.818423 22635 raft_consensus.cc:348] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.818650 22635 raft_consensus.cc:374] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:01.818763 22635 raft_consensus.cc:729] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:01.818980 22635 consensus_queue.cc:260] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:01.819870 22635 ts_tablet_manager.cc:1398] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:01.893221 22602 raft_consensus.cc:482] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:01.893422 22602 raft_consensus.cc:504] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.894483 22602 leader_election.cc:288] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:01.895141 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de41d6c6de0400397645c05a0fa17de" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:01.895462 21551 raft_consensus.cc:2454] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:01.895984 21749 leader_election.cc:302] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:01.896287 22602 raft_consensus.cc:2781] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:01.896415 22602 raft_consensus.cc:482] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:01.896536 22602 raft_consensus.cc:3037] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.898591 22602 raft_consensus.cc:504] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.899279 22602 leader_election.cc:288] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:01.899808 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de41d6c6de0400397645c05a0fa17de" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:01.900022 21551 raft_consensus.cc:3037] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.901754 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de41d6c6de0400397645c05a0fa17de" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:01.901710 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de41d6c6de0400397645c05a0fa17de" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:01.901971 21944 raft_consensus.cc:3037] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.902544 21551 raft_consensus.cc:2454] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:01.902964 21749 leader_election.cc:302] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:01.903349 22602 raft_consensus.cc:2781] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:01.903838 22602 raft_consensus.cc:686] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:01.904147 22602 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.904372 21944 raft_consensus.cc:2454] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:01.907166 21402 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:01.930941 22602 raft_consensus.cc:482] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:01.931277 22602 raft_consensus.cc:504] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.931988 22602 leader_election.cc:288] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:01.932668 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc49c050280346db88d1323e30513c80" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:01.932682 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc49c050280346db88d1323e30513c80" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:01.932927 21944 raft_consensus.cc:2454] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:01.932979 21551 raft_consensus.cc:2454] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:01.933382 21749 leader_election.cc:302] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters:
I20240502 08:20:01.933687 22602 raft_consensus.cc:2781] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:01.933825 22602 raft_consensus.cc:482] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:01.933939 22602 raft_consensus.cc:3037] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.936017 22602 raft_consensus.cc:504] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.936625 22602 leader_election.cc:288] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:01.937171 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc49c050280346db88d1323e30513c80" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:01.937378 21551 raft_consensus.cc:3037] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.937235 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc49c050280346db88d1323e30513c80" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:01.937566 21944 raft_consensus.cc:3037] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:01.939576 21944 raft_consensus.cc:2454] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:01.939580 21551 raft_consensus.cc:2454] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:01.940030 21749 leader_election.cc:302] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:01.940371 22602 raft_consensus.cc:2781] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:01.940539 22602 raft_consensus.cc:686] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:01.940846 22602 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:01.943881 21402 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:02.107504 22629 raft_consensus.cc:482] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:02.107724 22629 raft_consensus.cc:504] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:02.108469 22629 leader_election.cc:288] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:02.109254 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3898bfdb9c3d4a05af78f79fd2de61e2" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:02.109254 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3898bfdb9c3d4a05af78f79fd2de61e2" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:02.109520 21812 raft_consensus.cc:2454] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:02.109591 21551 raft_consensus.cc:2454] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:02.110162 21619 leader_election.cc:302] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:02.110572 22629 raft_consensus.cc:2781] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:02.110687 22629 raft_consensus.cc:482] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:02.110747 22629 raft_consensus.cc:3037] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:02.112865 22629 raft_consensus.cc:504] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:02.113458 22629 leader_election.cc:288] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:02.114080 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3898bfdb9c3d4a05af78f79fd2de61e2" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:02.114104 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3898bfdb9c3d4a05af78f79fd2de61e2" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:02.114284 21812 raft_consensus.cc:3037] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:02.114315 21551 raft_consensus.cc:3037] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:02.116364 21812 raft_consensus.cc:2454] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:02.116540 21551 raft_consensus.cc:2454] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:02.116961 21618 leader_election.cc:302] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:02.117372 22629 raft_consensus.cc:2781] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:02.117831 22629 raft_consensus.cc:686] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:02.118126 22629 consensus_queue.cc:237] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:02.121152 21402 catalog_manager.cc:5484] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:02.346848 22602 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:02.352233 22602 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:02.352358 22651 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:02.384357 22602 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:02.508945 22629 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:02.521970 22629 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:02.525127 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:02.525621 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:02.542924 22656 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 259.443 ms
time per row: 0.129722 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:20:03.928907 21402 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:03.939271 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.939831 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.939747 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.939831 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.940099 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.940259 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.944015 21924 tablet_service.cc:1467] Processing CreateTablet for tablet 80b3ba1d834a4314b63f336675845441 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.944449 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.948138 21661 tablet_service.cc:1467] Processing CreateTablet for tablet 80b3ba1d834a4314b63f336675845441 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.948714 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.949827 22724 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:03.951633 22724 tablet_bootstrap.cc:654] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.952797 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 80b3ba1d834a4314b63f336675845441 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.953375 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.954881 21922 tablet_service.cc:1467] Processing CreateTablet for tablet a20b2fd9f1ce488ba7793ed7ac268c16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.955367 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.955670 21531 tablet_service.cc:1467] Processing CreateTablet for tablet a20b2fd9f1ce488ba7793ed7ac268c16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.956090 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.960505 22723 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:03.961099 21662 tablet_service.cc:1467] Processing CreateTablet for tablet a20b2fd9f1ce488ba7793ed7ac268c16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:03.961549 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:03.963088 22723 tablet_bootstrap.cc:654] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.967396 22725 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:03.969841 22725 tablet_bootstrap.cc:654] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.970592 22727 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:03.972270 22724 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:03.972430 22724 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240502 08:20:03.972759 22727 tablet_bootstrap.cc:654] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.973307 22724 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.973546 22723 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:03.973554 22724 raft_consensus.cc:374] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.973649 22724 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.973908 22724 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.973718 22723 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 08:20:03.974813 22725 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:03.975037 22725 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:20:03.975164 22723 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.975399 22724 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:20:03.975438 22723 raft_consensus.cc:374] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.975539 22723 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.975728 22724 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:03.975833 22723 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.976357 22725 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.976665 22725 raft_consensus.cc:374] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.976817 22725 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.977123 22725 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:03.977735 22725 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:03.978147 22725 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:03.979271 22727 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:03.979496 22727 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:20:03.980512 22725 tablet_bootstrap.cc:654] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.980664 22727 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.980953 22727 raft_consensus.cc:374] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.981103 22727 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.981412 22727 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.982789 22727 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:03.983805 22724 tablet_bootstrap.cc:654] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.986889 22725 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:03.987120 22725 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 08:20:03.987849 22724 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:03.988035 22724 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:20:03.988054 22725 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.988339 22725 raft_consensus.cc:374] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.987352 22723 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20240502 08:20:03.988432 22725 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.988691 22725 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.988910 22724 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.989112 22723 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:03.989149 22724 raft_consensus.cc:374] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.989225 22724 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.989270 22725 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:03.989478 22724 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.990083 22724 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:03.990662 22724 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:03.992416 22723 tablet_bootstrap.cc:654] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.992435 22724 tablet_bootstrap.cc:654] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.995366 22723 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:03.995553 22723 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:20:03.996307 22723 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.996565 22723 raft_consensus.cc:374] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.996659 22723 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:03.996897 22723 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.997383 22723 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:03.997763 22723 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:03.998411 22724 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:03.998659 22724 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:20:03.999068 22723 tablet_bootstrap.cc:654] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:03.999544 22724 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:03.999773 22724 raft_consensus.cc:374] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:03.999887 22724 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:04.000134 22724 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.000726 22724 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:04.001086 22723 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:04.001227 22723 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:20:04.002063 22723 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.002292 22723 raft_consensus.cc:374] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:04.002391 22723 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:04.002660 22723 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.003294 22723 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:04.028748 22671 raft_consensus.cc:482] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:04.028954 22671 raft_consensus.cc:504] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.029690 22671 leader_election.cc:288] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:04.030566 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20b2fd9f1ce488ba7793ed7ac268c16" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:04.030810 21944 raft_consensus.cc:2454] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:04.031040 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20b2fd9f1ce488ba7793ed7ac268c16" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:04.031360 21619 leader_election.cc:302] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:04.031697 22671 raft_consensus.cc:2781] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:04.031819 22671 raft_consensus.cc:482] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:04.031769 21551 raft_consensus.cc:2454] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:04.031901 22671 raft_consensus.cc:3037] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.034967 22671 raft_consensus.cc:504] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.035706 22671 leader_election.cc:288] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:04.036381 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20b2fd9f1ce488ba7793ed7ac268c16" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:04.036474 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20b2fd9f1ce488ba7793ed7ac268c16" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:04.036710 21944 raft_consensus.cc:3037] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.036697 21551 raft_consensus.cc:3037] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.038738 21551 raft_consensus.cc:2454] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:04.039315 21619 leader_election.cc:302] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:04.039659 22671 raft_consensus.cc:2781] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:04.039754 21944 raft_consensus.cc:2454] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:04.040311 22671 raft_consensus.cc:686] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:04.040709 22671 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.043941 21402 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:04.047510 22731 raft_consensus.cc:482] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:04.047706 22731 raft_consensus.cc:504] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.049357 22731 leader_election.cc:288] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:04.066552 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b3ba1d834a4314b63f336675845441" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:04.066504 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b3ba1d834a4314b63f336675845441" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:04.066812 21682 raft_consensus.cc:2454] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:04.066848 21813 raft_consensus.cc:2454] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:04.067430 21880 leader_election.cc:302] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:04.067880 22731 raft_consensus.cc:2781] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:04.068039 22731 raft_consensus.cc:482] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:04.068143 22731 raft_consensus.cc:3037] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.070168 22731 raft_consensus.cc:504] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.070855 22731 leader_election.cc:288] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:04.071350 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b3ba1d834a4314b63f336675845441" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:04.071414 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b3ba1d834a4314b63f336675845441" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:04.071594 21813 raft_consensus.cc:3037] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.071624 21682 raft_consensus.cc:3037] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.073470 21813 raft_consensus.cc:2454] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:04.073483 21682 raft_consensus.cc:2454] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:04.073905 21880 leader_election.cc:302] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:04.074208 22731 raft_consensus.cc:2781] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:04.074632 22731 raft_consensus.cc:686] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Becoming Leader. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Running, Role: LEADER
I20240502 08:20:04.075091 22731 consensus_queue.cc:237] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:04.078711 21402 catalog_manager.cc:5484] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196). New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:04.150068 22671 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:04.150301 22671 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:04.150929 22671 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:04.151618 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:04.151661 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:04.151849 21551 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:04.151913 21944 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:04.152350 21619 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:04.152698 22671 raft_consensus.cc:2781] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:04.152830 22671 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:04.152941 22671 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.155128 22671 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:04.155747 22671 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:04.156201 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:04.156270 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:04.156404 21551 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.156492 21944 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:04.158453 21944 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:04.158468 21551 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:04.158941 21619 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:04.159312 22671 raft_consensus.cc:2781] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:04.159474 22671 raft_consensus.cc:686] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:04.159739 22671 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:04.162530 21402 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:04.456413 22671 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:04.463485 22671 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:04.510599 22731 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:04.516400 22743 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:04.570776 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:04.571158 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:04.579314 22746 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:20:04.709595 22671 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:04.713193 22671 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 347.85 ms
time per row: 0.173925 ms
I20240502 08:20:05.088523 21410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:05.098755 21792 tablet_service.cc:1467] Processing CreateTablet for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.099023 21662 tablet_service.cc:1467] Processing CreateTablet for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.099337 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.099485 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.099509 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 296ccf9be99545789bbb3ff4031d3170 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.099946 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296ccf9be99545789bbb3ff4031d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.100662 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 296ccf9be99545789bbb3ff4031d3170 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.101119 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296ccf9be99545789bbb3ff4031d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.103781 21531 tablet_service.cc:1467] Processing CreateTablet for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.104234 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.107404 22807 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:05.108366 21663 tablet_service.cc:1467] Processing CreateTablet for tablet cec56facd7f740aeb42bc7c32495dae3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.108490 21924 tablet_service.cc:1467] Processing CreateTablet for tablet cec56facd7f740aeb42bc7c32495dae3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.108845 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.108942 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.109305 22807 tablet_bootstrap.cc:654] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.111951 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 296ccf9be99545789bbb3ff4031d3170 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.112569 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296ccf9be99545789bbb3ff4031d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.119122 21792 tablet_service.cc:1467] Processing CreateTablet for tablet cec56facd7f740aeb42bc7c32495dae3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:05.119829 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:05.133796 22810 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:05.137550 22810 tablet_bootstrap.cc:654] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.140497 22809 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:05.140590 22807 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:05.140806 22807 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.018s
I20240502 08:20:05.141477 22811 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:05.141813 22807 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.142058 22807 raft_consensus.cc:374] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.142165 22807 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.142488 22807 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.143232 22807 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 08:20:05.143486 22811 tablet_bootstrap.cc:654] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.144165 22809 tablet_bootstrap.cc:654] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.145489 22807 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:05.145814 22810 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:05.145972 22810 ts_tablet_manager.cc:1367] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:20:05.147065 22810 raft_consensus.cc:348] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.147477 22810 raft_consensus.cc:374] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.147655 22810 raft_consensus.cc:729] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.148272 22810 consensus_queue.cc:260] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.149221 22810 ts_tablet_manager.cc:1398] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:20:05.149277 22807 tablet_bootstrap.cc:654] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.149626 22810 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:05.151667 22810 tablet_bootstrap.cc:654] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.153959 22811 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:05.154194 22811 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:20:05.155304 22811 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.155544 22811 raft_consensus.cc:374] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.155642 22811 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.155930 22811 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.156263 22809 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:05.156430 22809 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20240502 08:20:05.157030 22811 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:20:05.157392 22811 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:05.157567 22809 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.157784 22809 raft_consensus.cc:374] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.157883 22809 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.158149 22809 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.158982 22809 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 08:20:05.159555 22809 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:05.160111 22811 tablet_bootstrap.cc:654] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.160486 22737 raft_consensus.cc:482] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:05.160629 22737 raft_consensus.cc:504] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.161515 22737 leader_election.cc:288] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:05.161937 22809 tablet_bootstrap.cc:654] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.162503 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96a7174eb5442f2b0cc0cafc3cc4c80" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:05.163264 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96a7174eb5442f2b0cc0cafc3cc4c80" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:05.163595 21813 raft_consensus.cc:2454] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
W20240502 08:20:05.163656 21619 leader_election.cc:341] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:20:05.164122 21618 leader_election.cc:302] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters: cfff665685a14d8796aece26addab59c
I20240502 08:20:05.164448 22737 raft_consensus.cc:2781] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:05.164569 22807 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:05.164587 22737 raft_consensus.cc:482] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:05.164683 22737 raft_consensus.cc:3037] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.164757 22807 ts_tablet_manager.cc:1367] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.010s
I20240502 08:20:05.165771 22807 raft_consensus.cc:348] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.166086 22807 raft_consensus.cc:374] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.166242 22807 raft_consensus.cc:729] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.166689 22807 consensus_queue.cc:260] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.168550 22737 raft_consensus.cc:504] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.168926 22809 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:05.168934 22810 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:05.169121 22810 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 08:20:05.169137 22809 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240502 08:20:05.169898 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96a7174eb5442f2b0cc0cafc3cc4c80" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:05.169975 22809 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.170140 21813 raft_consensus.cc:3037] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.170192 22809 raft_consensus.cc:374] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.170261 22809 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.170178 22810 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.170429 22810 raft_consensus.cc:374] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.170526 22810 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.170850 22810 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.171614 22810 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:20:05.171533 22809 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.172130 22809 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:20:05.172155 22807 ts_tablet_manager.cc:1398] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240502 08:20:05.172979 21813 raft_consensus.cc:2454] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:05.173650 21618 leader_election.cc:302] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:05.174013 22671 raft_consensus.cc:2781] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:05.174175 22671 raft_consensus.cc:686] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:05.174482 22671 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:05.177412 22737 leader_election.cc:288] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:05.178710 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96a7174eb5442f2b0cc0cafc3cc4c80" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:05.179004 21551 raft_consensus.cc:3037] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.183729 22807 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:05.186311 22807 tablet_bootstrap.cc:654] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:05.189146 22811 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:05.189352 22811 ts_tablet_manager.cc:1367] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.010s
I20240502 08:20:05.190147 22811 raft_consensus.cc:348] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.190397 22811 raft_consensus.cc:374] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.190482 22811 raft_consensus.cc:729] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.190805 22811 consensus_queue.cc:260] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.191430 22811 ts_tablet_manager.cc:1398] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:05.197966 21551 raft_consensus.cc:2454] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:05.205467 21407 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:05.206414 22807 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:05.208269 22807 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.006s
I20240502 08:20:05.209172 22807 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.209456 22807 raft_consensus.cc:374] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:05.209568 22807 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:05.210089 22807 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.210649 22807 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:05.331521 22743 raft_consensus.cc:482] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:05.331826 22743 raft_consensus.cc:504] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.332563 22743 leader_election.cc:288] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:05.333051 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec56facd7f740aeb42bc7c32495dae3" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:05.333267 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec56facd7f740aeb42bc7c32495dae3" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:05.333348 21682 raft_consensus.cc:2454] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:05.333529 21813 raft_consensus.cc:2454] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:05.333905 21880 leader_election.cc:302] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:05.334208 22731 raft_consensus.cc:2781] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:05.334290 22731 raft_consensus.cc:482] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:05.334405 22731 raft_consensus.cc:3037] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.336258 22731 raft_consensus.cc:504] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.336833 22731 leader_election.cc:288] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:05.337433 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec56facd7f740aeb42bc7c32495dae3" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:05.337433 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec56facd7f740aeb42bc7c32495dae3" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:05.337684 21813 raft_consensus.cc:3037] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.337728 21682 raft_consensus.cc:3037] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.340669 21682 raft_consensus.cc:2454] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:05.340662 21813 raft_consensus.cc:2454] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:05.341195 21880 leader_election.cc:302] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:05.341539 22731 raft_consensus.cc:2781] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:05.341773 22731 raft_consensus.cc:686] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Becoming Leader. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Running, Role: LEADER
I20240502 08:20:05.342067 22731 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.344934 21407 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196). New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:05.411134 22791 raft_consensus.cc:482] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:05.411473 22791 raft_consensus.cc:504] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.412254 22791 leader_election.cc:288] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:05.412837 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296ccf9be99545789bbb3ff4031d3170" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:05.412935 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296ccf9be99545789bbb3ff4031d3170" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:05.413082 21551 raft_consensus.cc:2454] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:05.413223 21943 raft_consensus.cc:2454] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:05.413563 21749 leader_election.cc:302] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:05.413916 22791 raft_consensus.cc:2781] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:05.414024 22791 raft_consensus.cc:482] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:05.414113 22791 raft_consensus.cc:3037] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.416141 22791 raft_consensus.cc:504] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.416710 22791 leader_election.cc:288] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:05.417291 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296ccf9be99545789bbb3ff4031d3170" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:05.417351 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296ccf9be99545789bbb3ff4031d3170" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:05.417498 21551 raft_consensus.cc:3037] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.417618 21943 raft_consensus.cc:3037] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:05.419600 21551 raft_consensus.cc:2454] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:05.419723 21943 raft_consensus.cc:2454] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:05.420157 21749 leader_election.cc:302] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:05.420466 22791 raft_consensus.cc:2781] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:05.420879 22791 raft_consensus.cc:686] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:05.421196 22791 consensus_queue.cc:237] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:05.424387 21407 catalog_manager.cc:5484] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:05.654765 22671 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:05.660104 22671 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:05.807173 22731 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:05.813683 22731 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:05.818451 22824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:05.818944 22824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:05.827088 22824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 08:20:05.880164 22791 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:05.886643 22791 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.468 ms
time per row: 0.115234 ms
I20240502 08:20:06.246965 21410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:06.257458 21662 tablet_service.cc:1467] Processing CreateTablet for tablet 599d5781f1e24ae9ad425a34d88c6b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.257678 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 599d5781f1e24ae9ad425a34d88c6b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.257714 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 599d5781f1e24ae9ad425a34d88c6b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.257966 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599d5781f1e24ae9ad425a34d88c6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.258132 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599d5781f1e24ae9ad425a34d88c6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.258150 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599d5781f1e24ae9ad425a34d88c6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.260298 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 59f5d97e6e0a4755be41699246907e55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.260731 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.260901 21924 tablet_service.cc:1467] Processing CreateTablet for tablet 1648586b889b4a1e9ac34481cec6257a (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.261351 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1648586b889b4a1e9ac34481cec6257a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.262239 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 59f5d97e6e0a4755be41699246907e55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.262658 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.266340 21532 tablet_service.cc:1467] Processing CreateTablet for tablet 59f5d97e6e0a4755be41699246907e55 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.266496 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 1648586b889b4a1e9ac34481cec6257a (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.266830 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.267342 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1648586b889b4a1e9ac34481cec6257a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.272334 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 1648586b889b4a1e9ac34481cec6257a (DEFAULT_TABLE table=rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:06.272902 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1648586b889b4a1e9ac34481cec6257a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:06.272491 22890 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:06.274993 22890 tablet_bootstrap.cc:654] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.283483 22890 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:06.283690 22890 ts_tablet_manager.cc:1367] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:20:06.284588 22890 raft_consensus.cc:348] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.284845 22890 raft_consensus.cc:374] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.284937 22890 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.285202 22890 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.285964 22890 ts_tablet_manager.cc:1398] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:06.291116 22893 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:06.294988 22893 tablet_bootstrap.cc:654] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.299424 22890 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:06.301868 22890 tablet_bootstrap.cc:654] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.304262 22895 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:06.305949 22893 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:06.305936 22890 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:06.306138 22890 ts_tablet_manager.cc:1367] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:20:06.306298 22893 ts_tablet_manager.cc:1367] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 08:20:06.307134 22895 tablet_bootstrap.cc:654] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.307053 22890 raft_consensus.cc:348] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.307142 22893 raft_consensus.cc:348] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.307353 22890 raft_consensus.cc:374] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.307428 22890 raft_consensus.cc:729] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.307436 22893 raft_consensus.cc:374] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.307539 22893 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.307718 22890 consensus_queue.cc:260] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.307830 22893 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.308338 22890 ts_tablet_manager.cc:1398] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:20:06.309940 22891 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:06.310767 22671 raft_consensus.cc:482] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:06.310971 22671 raft_consensus.cc:504] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.311733 22671 leader_election.cc:288] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:06.312487 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:06.313149 22891 tablet_bootstrap.cc:654] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
W20240502 08:20:06.313494 21618 leader_election.cc:341] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:20:06.313988 22893 ts_tablet_manager.cc:1398] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:20:06.314388 22895 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:06.314412 22893 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:06.314671 22895 ts_tablet_manager.cc:1367] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240502 08:20:06.314998 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:06.315284 21552 raft_consensus.cc:2454] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:06.315802 21619 leader_election.cc:302] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters: 13e764b27ed449c09aee719cdb2597f8
I20240502 08:20:06.316321 22671 raft_consensus.cc:2781] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:06.316434 22671 raft_consensus.cc:482] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:06.316535 22671 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.317234 22893 tablet_bootstrap.cc:654] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.317860 22895 raft_consensus.cc:348] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.318171 22895 raft_consensus.cc:374] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.318316 22895 raft_consensus.cc:729] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.318708 22895 consensus_queue.cc:260] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.319439 22895 ts_tablet_manager.cc:1398] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:20:06.319912 22895 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:06.321833 22895 tablet_bootstrap.cc:654] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.329877 22671 raft_consensus.cc:504] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.331837 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
W20240502 08:20:06.332785 21618 leader_election.cc:341] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:20:06.334136 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:06.334462 21552 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.338840 21552 raft_consensus.cc:2454] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:06.339597 21619 leader_election.cc:302] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters: 13e764b27ed449c09aee719cdb2597f8
I20240502 08:20:06.340217 22885 raft_consensus.cc:2781] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:06.340394 22885 raft_consensus.cc:686] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:06.340816 22885 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.344738 22671 leader_election.cc:288] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:06.353428 22891 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:06.353796 22891 ts_tablet_manager.cc:1367] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.034s
I20240502 08:20:06.354390 21404 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:20:06.355332 22891 raft_consensus.cc:348] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.355813 22891 raft_consensus.cc:374] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.355981 22891 raft_consensus.cc:729] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.356465 22891 consensus_queue.cc:260] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.357784 22891 ts_tablet_manager.cc:1398] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:20:06.358726 22893 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:06.358943 22893 ts_tablet_manager.cc:1367] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.018s
I20240502 08:20:06.359526 22895 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:06.359781 22895 ts_tablet_manager.cc:1367] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.014s
I20240502 08:20:06.359839 22891 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:06.360227 22893 raft_consensus.cc:348] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.360479 22893 raft_consensus.cc:374] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.360575 22893 raft_consensus.cc:729] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.360855 22893 consensus_queue.cc:260] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.361943 22891 tablet_bootstrap.cc:654] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.361526 22893 ts_tablet_manager.cc:1398] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:06.362538 22893 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:06.362807 22895 raft_consensus.cc:348] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.363116 22895 raft_consensus.cc:374] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.363260 22895 raft_consensus.cc:729] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.363672 22895 consensus_queue.cc:260] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.364385 22893 tablet_bootstrap.cc:654] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:06.369266 22895 ts_tablet_manager.cc:1398] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:20:06.371682 22893 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:06.371909 22893 ts_tablet_manager.cc:1367] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 08:20:06.372113 22891 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:06.372385 22891 ts_tablet_manager.cc:1367] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240502 08:20:06.372874 22893 raft_consensus.cc:348] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.373114 22893 raft_consensus.cc:374] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.373262 22893 raft_consensus.cc:729] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.373183 22891 raft_consensus.cc:348] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.373430 22891 raft_consensus.cc:374] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:06.373512 22891 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:06.373569 22893 consensus_queue.cc:260] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.373777 22891 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.374181 22893 ts_tablet_manager.cc:1398] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:06.374476 22891 ts_tablet_manager.cc:1398] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:20:06.523550 22630 raft_consensus.cc:482] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:06.523757 22630 raft_consensus.cc:504] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.524698 22630 leader_election.cc:288] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:06.525291 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f5d97e6e0a4755be41699246907e55" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:06.525640 21943 raft_consensus.cc:2454] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:06.526214 21488 leader_election.cc:302] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:06.526551 22630 raft_consensus.cc:2781] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:06.526679 22630 raft_consensus.cc:482] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:06.526785 22630 raft_consensus.cc:3037] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.529219 22630 raft_consensus.cc:504] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.530232 22630 leader_election.cc:288] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:06.530664 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f5d97e6e0a4755be41699246907e55" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:06.530879 21943 raft_consensus.cc:3037] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.531939 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f5d97e6e0a4755be41699246907e55" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:06.532006 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f5d97e6e0a4755be41699246907e55" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:06.532224 21813 raft_consensus.cc:2454] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:06.533317 21943 raft_consensus.cc:2454] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:20:06.533807 21488 leader_election.cc:302] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c; no voters: 13e764b27ed449c09aee719cdb2597f8
I20240502 08:20:06.534160 22630 raft_consensus.cc:2781] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:06.534583 22630 raft_consensus.cc:686] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Becoming Leader. State: Replica: cfff665685a14d8796aece26addab59c, State: Running, Role: LEADER
I20240502 08:20:06.534950 22630 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:06.537943 21408 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c reported cstate change: term changed from 0 to 1, leader changed from <none> to cfff665685a14d8796aece26addab59c (127.8.68.193). New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:06.594380 22884 raft_consensus.cc:482] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:06.594609 22884 raft_consensus.cc:504] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:06.595428 22884 leader_election.cc:288] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:06.595866 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:06.595953 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:06.596261 21552 raft_consensus.cc:2379] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 13e764b27ed449c09aee719cdb2597f8 in current term 1: Already voted for candidate 60677121510e47a5b51d1eb22f9aa238 in this term.
I20240502 08:20:06.596707 21749 leader_election.cc:302] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8; no voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c
I20240502 08:20:06.597004 22884 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.599524 22884 raft_consensus.cc:2726] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:20:06.682773 22885 raft_consensus.cc:482] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:06.682979 22885 raft_consensus.cc:504] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.683760 22885 leader_election.cc:288] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:06.684499 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1648586b889b4a1e9ac34481cec6257a" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:06.684468 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1648586b889b4a1e9ac34481cec6257a" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:06.684824 21552 raft_consensus.cc:2454] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:06.684846 21943 raft_consensus.cc:2454] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:06.685326 21619 leader_election.cc:302] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:06.685655 22885 raft_consensus.cc:2781] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:06.685756 22885 raft_consensus.cc:482] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:06.685878 22885 raft_consensus.cc:3037] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.689031 22885 raft_consensus.cc:504] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.689778 22885 leader_election.cc:288] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:06.690511 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1648586b889b4a1e9ac34481cec6257a" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:06.690770 21943 raft_consensus.cc:3037] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.690725 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1648586b889b4a1e9ac34481cec6257a" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:06.690956 21552 raft_consensus.cc:3037] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:06.693823 21943 raft_consensus.cc:2454] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:06.693987 21552 raft_consensus.cc:2454] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:06.694523 21619 leader_election.cc:302] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:06.694962 22885 raft_consensus.cc:2781] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:06.695273 22885 raft_consensus.cc:686] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:06.695577 22885 consensus_queue.cc:237] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:06.698808 21408 catalog_manager.cc:5484] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:06.739305 22885 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:06.747090 22885 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:07.101001 22885 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:07.101639 22630 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:07.107424 22737 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:07.110654 21813 raft_consensus.cc:3037] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.116042 22917 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:07.142274 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:07.142625 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:07.150331 22914 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 257.859 ms
time per row: 0.12893 ms
I20240502 08:20:07.568250 21405 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:07.578701 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.578755 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.579229 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.580288 21792 tablet_service.cc:1467] Processing CreateTablet for tablet b88c4a1a36424086aad89b7585c17960 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.580341 21530 tablet_service.cc:1467] Processing CreateTablet for tablet e0241228bd204ca2a8ad2c57b6eeb543 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.580739 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0241228bd204ca2a8ad2c57b6eeb543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.580741 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.581897 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.582362 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.584076 21662 tablet_service.cc:1467] Processing CreateTablet for tablet b88c4a1a36424086aad89b7585c17960 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.584167 21661 tablet_service.cc:1467] Processing CreateTablet for tablet e0241228bd204ca2a8ad2c57b6eeb543 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.584473 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.584663 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0241228bd204ca2a8ad2c57b6eeb543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.579221 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.580292 21924 tablet_service.cc:1467] Processing CreateTablet for tablet b88c4a1a36424086aad89b7585c17960 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.592046 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.594087 22972 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:07.596307 22972 tablet_bootstrap.cc:654] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.596091 22973 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:07.598592 22973 tablet_bootstrap.cc:654] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.604461 21791 tablet_service.cc:1467] Processing CreateTablet for tablet e0241228bd204ca2a8ad2c57b6eeb543 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:07.604676 22976 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:07.606580 22976 tablet_bootstrap.cc:654] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.608445 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0241228bd204ca2a8ad2c57b6eeb543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:07.608747 22973 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:07.608991 22973 ts_tablet_manager.cc:1367] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240502 08:20:07.609930 22973 raft_consensus.cc:348] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.610244 22973 raft_consensus.cc:374] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.610404 22973 raft_consensus.cc:729] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.610735 22973 consensus_queue.cc:260] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.611582 22973 ts_tablet_manager.cc:1398] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:07.614621 22978 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:07.607376 22972 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:07.615424 22972 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.010s
I20240502 08:20:07.617267 22978 tablet_bootstrap.cc:654] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.617647 22972 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.617890 22972 raft_consensus.cc:374] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.618009 22972 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.618271 22972 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.620074 22972 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:20:07.624261 22972 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:07.626711 22972 tablet_bootstrap.cc:654] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.628055 22976 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:07.628224 22976 ts_tablet_manager.cc:1367] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240502 08:20:07.628897 22976 raft_consensus.cc:348] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.629073 22976 raft_consensus.cc:374] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.629171 22976 raft_consensus.cc:729] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.629412 22976 consensus_queue.cc:260] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.630700 22976 ts_tablet_manager.cc:1398] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:07.631006 22976 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:07.633730 22976 tablet_bootstrap.cc:654] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.636265 22978 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:07.636512 22978 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20240502 08:20:07.636905 22972 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:07.637086 22972 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240502 08:20:07.637423 22978 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.637651 22978 raft_consensus.cc:374] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.637773 22978 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.637898 22972 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.638027 22978 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.638144 22972 raft_consensus.cc:374] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.638247 22972 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.638486 22972 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.638660 22978 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:07.638981 22978 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:07.639101 22972 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:07.639449 22972 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:07.640921 22978 tablet_bootstrap.cc:654] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.641048 22972 tablet_bootstrap.cc:654] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.641901 22976 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:07.642139 22976 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240502 08:20:07.643131 22976 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.643491 22976 raft_consensus.cc:374] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.643623 22976 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.643744 22972 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:07.644099 22976 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.644989 22976 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:20:07.645452 22976 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:07.647518 22972 ts_tablet_manager.cc:1367] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:20:07.647946 22976 tablet_bootstrap.cc:654] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:07.648380 22972 raft_consensus.cc:348] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.648633 22972 raft_consensus.cc:374] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.648743 22972 raft_consensus.cc:729] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.650012 22972 consensus_queue.cc:260] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.650652 22972 ts_tablet_manager.cc:1398] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:07.651096 22976 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:07.651270 22976 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 08:20:07.652114 22976 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.652338 22976 raft_consensus.cc:374] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.652465 22976 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.652745 22976 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.653375 22978 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:07.653437 22976 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:20:07.653611 22978 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 08:20:07.654417 22978 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.654608 22978 raft_consensus.cc:374] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:07.654711 22978 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:07.654951 22978 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.655511 22978 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:07.681578 22917 raft_consensus.cc:482] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:07.681846 22917 raft_consensus.cc:504] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.682626 22917 leader_election.cc:288] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:07.683229 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0241228bd204ca2a8ad2c57b6eeb543" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:07.683310 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0241228bd204ca2a8ad2c57b6eeb543" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:07.683562 21682 raft_consensus.cc:2454] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:07.683578 21813 raft_consensus.cc:2454] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:07.684120 21487 leader_election.cc:302] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:07.684458 22917 raft_consensus.cc:2781] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:07.684552 22917 raft_consensus.cc:482] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:07.684659 22917 raft_consensus.cc:3037] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.687091 22917 raft_consensus.cc:504] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.688206 22917 leader_election.cc:288] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:07.689771 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0241228bd204ca2a8ad2c57b6eeb543" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:07.689993 21813 raft_consensus.cc:3037] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.691499 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0241228bd204ca2a8ad2c57b6eeb543" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:07.691763 21682 raft_consensus.cc:3037] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.692643 21813 raft_consensus.cc:2454] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:20:07.693207 21487 leader_election.cc:302] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:07.693533 22917 raft_consensus.cc:2781] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:07.693714 22917 raft_consensus.cc:686] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Becoming Leader. State: Replica: cfff665685a14d8796aece26addab59c, State: Running, Role: LEADER
I20240502 08:20:07.694021 22917 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:07.694622 21682 raft_consensus.cc:2454] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:20:07.697696 21405 catalog_manager.cc:5484] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c reported cstate change: term changed from 0 to 1, leader changed from <none> to cfff665685a14d8796aece26addab59c (127.8.68.193). New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:07.725010 22885 raft_consensus.cc:482] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:07.725267 22885 raft_consensus.cc:504] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.725973 22885 leader_election.cc:288] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:07.726760 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88c4a1a36424086aad89b7585c17960" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:07.726754 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88c4a1a36424086aad89b7585c17960" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:07.727041 21943 raft_consensus.cc:2454] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:07.727197 21813 raft_consensus.cc:2454] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:07.727672 21619 leader_election.cc:302] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:07.728003 22885 raft_consensus.cc:2781] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:07.728103 22885 raft_consensus.cc:482] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:07.728219 22885 raft_consensus.cc:3037] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.730350 22885 raft_consensus.cc:504] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.731138 22885 leader_election.cc:288] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:07.731842 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88c4a1a36424086aad89b7585c17960" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:07.731879 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88c4a1a36424086aad89b7585c17960" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:07.732081 21813 raft_consensus.cc:3037] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.732088 21943 raft_consensus.cc:3037] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.733552 22731 raft_consensus.cc:482] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:07.733912 22731 raft_consensus.cc:504] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.734735 22731 leader_election.cc:288] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:07.735110 21813 raft_consensus.cc:2454] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:07.735369 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f5c2dfcd974ee2b8720e9eb052c3f7" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:07.735572 21943 raft_consensus.cc:2454] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:07.735602 21812 raft_consensus.cc:2454] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:07.736008 21618 leader_election.cc:302] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:07.736099 21879 leader_election.cc:302] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters:
I20240502 08:20:07.736362 22731 raft_consensus.cc:2781] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:07.736486 22731 raft_consensus.cc:482] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:07.736589 22731 raft_consensus.cc:3037] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.736593 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f5c2dfcd974ee2b8720e9eb052c3f7" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:07.736812 21682 raft_consensus.cc:2454] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:07.737090 22885 raft_consensus.cc:2781] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:07.737260 22885 raft_consensus.cc:686] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:07.737543 22885 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.739295 22731 raft_consensus.cc:504] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.739913 22731 leader_election.cc:288] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:07.740357 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f5c2dfcd974ee2b8720e9eb052c3f7" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:07.740463 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f5c2dfcd974ee2b8720e9eb052c3f7" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:07.740553 21682 raft_consensus.cc:3037] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.740698 21812 raft_consensus.cc:3037] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:07.741276 21405 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:07.742681 21812 raft_consensus.cc:2454] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:07.743146 21879 leader_election.cc:302] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters:
I20240502 08:20:07.743484 22731 raft_consensus.cc:2781] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:07.744062 22731 raft_consensus.cc:686] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Becoming Leader. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Running, Role: LEADER
I20240502 08:20:07.744382 22731 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:07.747191 21405 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196). New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:07.765250 21682 raft_consensus.cc:2454] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
W20240502 08:20:08.144234 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:08.144589 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:08.152678 22990 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:20:08.153728 22731 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:08.173518 23002 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:08.179759 22917 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:08.194101 22918 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 08:20:08.246197 22885 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:08.252346 23023 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 262.083 ms
time per row: 0.131041 ms
I20240502 08:20:08.605620 21407 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:08.615789 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 182a6f2df91242cfa10f05ef788a19cb (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.615942 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 182a6f2df91242cfa10f05ef788a19cb (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.616333 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.616448 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.617177 21532 tablet_service.cc:1467] Processing CreateTablet for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.617548 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.618091 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 852613f14b324b38b5eecc29d4c602cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.618573 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852613f14b324b38b5eecc29d4c602cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.618834 21662 tablet_service.cc:1467] Processing CreateTablet for tablet 182a6f2df91242cfa10f05ef788a19cb (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.619293 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.623920 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 852613f14b324b38b5eecc29d4c602cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.624410 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852613f14b324b38b5eecc29d4c602cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.627884 21661 tablet_service.cc:1467] Processing CreateTablet for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.628506 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.632592 21924 tablet_service.cc:1467] Processing CreateTablet for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.633234 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.637331 23055 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:08.640516 23055 tablet_bootstrap.cc:654] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.632619 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 852613f14b324b38b5eecc29d4c602cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:08.641280 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852613f14b324b38b5eecc29d4c602cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:08.631969 23054 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:08.647719 23056 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:08.647842 23054 tablet_bootstrap.cc:654] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.651477 23056 tablet_bootstrap.cc:654] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.656927 23058 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:08.659094 23058 tablet_bootstrap.cc:654] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.659812 23054 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:08.660034 23054 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 08:20:08.661180 23054 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.661489 23054 raft_consensus.cc:374] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.661612 23054 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.661830 23055 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:08.661872 23054 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.662057 23055 ts_tablet_manager.cc:1367] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.004s
I20240502 08:20:08.662915 23055 raft_consensus.cc:348] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.663029 23054 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:20:08.663127 23055 raft_consensus.cc:374] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.663218 23055 raft_consensus.cc:729] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.663461 23055 consensus_queue.cc:260] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.664685 23056 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:08.664963 23056 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.010s
I20240502 08:20:08.666400 23056 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.666779 23056 raft_consensus.cc:374] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.666944 23056 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.667341 23056 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.668220 23056 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:08.669898 23056 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:08.669983 23055 ts_tablet_manager.cc:1398] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:20:08.671748 23058 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:08.671876 23056 tablet_bootstrap.cc:654] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.672256 23054 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:08.672072 23058 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 08:20:08.673776 23058 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.674141 23058 raft_consensus.cc:374] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.674270 23058 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.674638 23058 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.675379 23054 tablet_bootstrap.cc:654] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.675560 23058 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:08.675997 23058 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:08.678318 23058 tablet_bootstrap.cc:654] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.685282 23054 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:08.685585 23054 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240502 08:20:08.686259 23054 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.686470 23054 raft_consensus.cc:374] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.686561 23054 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.686924 23054 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.688145 23058 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:08.688383 23058 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 08:20:08.689851 23058 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.690274 23058 raft_consensus.cc:374] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.690375 23058 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.690768 23058 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.690912 23054 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:20:08.691584 23058 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:20:08.692384 23054 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:08.694550 23054 tablet_bootstrap.cc:654] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.696775 23056 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:08.696991 23056 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.010s
I20240502 08:20:08.697886 23056 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.698146 23056 raft_consensus.cc:374] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.698237 23056 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.698571 23056 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.699211 23056 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:08.699681 23056 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:08.700341 23054 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:08.700510 23054 ts_tablet_manager.cc:1367] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240502 08:20:08.701359 23054 raft_consensus.cc:348] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.701570 23056 tablet_bootstrap.cc:654] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:08.701567 23054 raft_consensus.cc:374] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.701658 23054 raft_consensus.cc:729] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.701897 23054 consensus_queue.cc:260] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.703845 23054 ts_tablet_manager.cc:1398] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:20:08.721022 23056 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:08.721377 23056 ts_tablet_manager.cc:1367] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.014s
I20240502 08:20:08.723009 23056 raft_consensus.cc:348] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.723219 23056 raft_consensus.cc:374] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:08.723356 23056 raft_consensus.cc:729] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:08.723639 23056 consensus_queue.cc:260] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.724298 23056 ts_tablet_manager.cc:1398] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:08.737500 23040 raft_consensus.cc:482] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:08.737726 23040 raft_consensus.cc:504] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.738590 23040 leader_election.cc:288] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:08.739372 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182a6f2df91242cfa10f05ef788a19cb" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:08.739378 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182a6f2df91242cfa10f05ef788a19cb" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:08.739656 21552 raft_consensus.cc:2454] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:08.739666 21943 raft_consensus.cc:2454] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:08.740231 21619 leader_election.cc:302] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:08.740617 23040 raft_consensus.cc:2781] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:08.740759 23040 raft_consensus.cc:482] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:08.740855 23040 raft_consensus.cc:3037] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.743302 23040 raft_consensus.cc:504] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.743899 23040 leader_election.cc:288] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:08.744534 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182a6f2df91242cfa10f05ef788a19cb" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:08.744591 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182a6f2df91242cfa10f05ef788a19cb" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:08.744828 21943 raft_consensus.cc:3037] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.744817 21552 raft_consensus.cc:3037] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.747821 21943 raft_consensus.cc:2454] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:08.747850 21552 raft_consensus.cc:2454] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:08.748379 21619 leader_election.cc:302] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:08.748735 23040 raft_consensus.cc:2781] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:08.748961 23040 raft_consensus.cc:686] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:08.749293 23040 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.752430 21406 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:08.756368 22988 raft_consensus.cc:482] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:08.756608 22988 raft_consensus.cc:504] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.757406 22988 leader_election.cc:288] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:08.758491 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bb1e3e2b9d4e31bf93ac1ce08a7833" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:08.758744 21683 raft_consensus.cc:2454] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:08.759187 21880 leader_election.cc:302] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:08.759492 22988 raft_consensus.cc:2781] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:08.759613 22988 raft_consensus.cc:482] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:08.759708 22988 raft_consensus.cc:3037] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.762442 22988 raft_consensus.cc:504] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.763221 22988 leader_election.cc:288] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:08.763772 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bb1e3e2b9d4e31bf93ac1ce08a7833" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:08.763958 21683 raft_consensus.cc:3037] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.764623 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bb1e3e2b9d4e31bf93ac1ce08a7833" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:08.764722 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bb1e3e2b9d4e31bf93ac1ce08a7833" candidate_uuid: "0f655ab8f20d4ba482c3173f9594b98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:08.764897 21552 raft_consensus.cc:2454] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 0.
I20240502 08:20:08.766242 21683 raft_consensus.cc:2454] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f655ab8f20d4ba482c3173f9594b98b in term 1.
I20240502 08:20:08.766695 21880 leader_election.cc:302] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters: cfff665685a14d8796aece26addab59c
I20240502 08:20:08.767024 22988 raft_consensus.cc:2781] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:08.767187 22988 raft_consensus.cc:686] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Becoming Leader. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Running, Role: LEADER
I20240502 08:20:08.767539 22988 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:08.770798 21406 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196). New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:08.844487 22979 raft_consensus.cc:482] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:08.844695 22979 raft_consensus.cc:504] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.845434 22979 leader_election.cc:288] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:08.846084 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852613f14b324b38b5eecc29d4c602cd" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:08.846135 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852613f14b324b38b5eecc29d4c602cd" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:08.846328 21943 raft_consensus.cc:2454] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:08.846385 21552 raft_consensus.cc:2454] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:08.846812 21749 leader_election.cc:302] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters:
I20240502 08:20:08.847149 22979 raft_consensus.cc:2781] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:08.847317 22979 raft_consensus.cc:482] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:08.847400 22979 raft_consensus.cc:3037] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.849443 22979 raft_consensus.cc:504] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.849989 22979 leader_election.cc:288] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:08.850647 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852613f14b324b38b5eecc29d4c602cd" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:08.850673 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "852613f14b324b38b5eecc29d4c602cd" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:08.850868 21943 raft_consensus.cc:3037] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.850859 21552 raft_consensus.cc:3037] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:08.852852 21943 raft_consensus.cc:2454] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:08.852931 21552 raft_consensus.cc:2454] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:08.853379 21749 leader_election.cc:302] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters:
I20240502 08:20:08.853689 22979 raft_consensus.cc:2781] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:08.854152 22979 raft_consensus.cc:686] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:08.854470 22979 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:08.857592 21406 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:20:09.190958 23040 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:09.215080 23040 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:09.231901 21552 raft_consensus.cc:3037] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:09.236642 22988 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:09.237766 22988 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:09.299064 23072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:09.299443 23072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:09.307235 23072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 08:20:09.408601 22979 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:09.427170 23103 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 289.758 ms
time per row: 0.144879 ms
I20240502 08:20:09.769644 21403 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:09.779206 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfc3eaed583458b8381d4e7b4d36aef (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.779749 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfc3eaed583458b8381d4e7b4d36aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.779619 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfc3eaed583458b8381d4e7b4d36aef (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.779863 21662 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfc3eaed583458b8381d4e7b4d36aef (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.780108 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfc3eaed583458b8381d4e7b4d36aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.780330 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfc3eaed583458b8381d4e7b4d36aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.782873 21792 tablet_service.cc:1467] Processing CreateTablet for tablet c4862ea2f67c43a28689b464d8ec96e6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.783355 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4862ea2f67c43a28689b464d8ec96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.783345 21793 tablet_service.cc:1467] Processing CreateTablet for tablet efec2cc693bd4dd992d0bec22fca5419 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.783768 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efec2cc693bd4dd992d0bec22fca5419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.789301 21530 tablet_service.cc:1467] Processing CreateTablet for tablet c4862ea2f67c43a28689b464d8ec96e6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.789423 21923 tablet_service.cc:1467] Processing CreateTablet for tablet c4862ea2f67c43a28689b464d8ec96e6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.789944 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4862ea2f67c43a28689b464d8ec96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.790016 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4862ea2f67c43a28689b464d8ec96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.791801 21924 tablet_service.cc:1467] Processing CreateTablet for tablet efec2cc693bd4dd992d0bec22fca5419 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.792212 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efec2cc693bd4dd992d0bec22fca5419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.792094 21661 tablet_service.cc:1467] Processing CreateTablet for tablet efec2cc693bd4dd992d0bec22fca5419 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:09.792526 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efec2cc693bd4dd992d0bec22fca5419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:09.794739 23148 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:09.797842 23148 tablet_bootstrap.cc:654] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.804066 23150 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:09.808280 23150 tablet_bootstrap.cc:654] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.813189 23148 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:09.813380 23148 ts_tablet_manager.cc:1367] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240502 08:20:09.814335 23148 raft_consensus.cc:348] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.814713 23148 raft_consensus.cc:374] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.814867 23148 raft_consensus.cc:729] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.815166 23148 consensus_queue.cc:260] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.816793 23148 ts_tablet_manager.cc:1398] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:09.817126 23148 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:09.818890 23148 tablet_bootstrap.cc:654] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.823002 23150 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:09.823321 23150 ts_tablet_manager.cc:1367] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.000s
I20240502 08:20:09.824007 23148 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:09.824241 23152 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:09.824304 23148 ts_tablet_manager.cc:1367] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 08:20:09.824388 23150 raft_consensus.cc:348] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.824654 23150 raft_consensus.cc:374] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.824800 23150 raft_consensus.cc:729] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.825100 23150 consensus_queue.cc:260] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.825888 23150 ts_tablet_manager.cc:1398] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:09.826333 23150 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:09.827019 23148 raft_consensus.cc:348] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.827461 23148 raft_consensus.cc:374] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.827612 23148 raft_consensus.cc:729] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.828054 23150 tablet_bootstrap.cc:654] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.828047 23148 consensus_queue.cc:260] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.832329 23152 tablet_bootstrap.cc:654] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.834108 23148 ts_tablet_manager.cc:1398] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:20:09.834573 23148 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:09.834933 23147 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:09.836726 23148 tablet_bootstrap.cc:654] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.837159 23147 tablet_bootstrap.cc:654] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.837540 23150 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:09.837754 23150 ts_tablet_manager.cc:1367] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240502 08:20:09.838884 23150 raft_consensus.cc:348] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.839329 23150 raft_consensus.cc:374] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.839468 23150 raft_consensus.cc:729] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.839890 23150 consensus_queue.cc:260] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.840792 23150 ts_tablet_manager.cc:1398] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:09.842598 23152 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:09.842810 23152 ts_tablet_manager.cc:1367] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240502 08:20:09.843567 23148 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:09.843725 23147 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:09.843772 23148 ts_tablet_manager.cc:1367] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240502 08:20:09.843966 23147 ts_tablet_manager.cc:1367] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:20:09.843989 23152 raft_consensus.cc:348] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.844324 23152 raft_consensus.cc:374] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.844465 23152 raft_consensus.cc:729] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.844591 23148 raft_consensus.cc:348] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.844818 23148 raft_consensus.cc:374] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.844936 23148 raft_consensus.cc:729] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.844928 23147 raft_consensus.cc:348] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.844861 23152 consensus_queue.cc:260] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.845166 23147 raft_consensus.cc:374] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.845268 23147 raft_consensus.cc:729] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.845206 23148 consensus_queue.cc:260] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.845572 23147 consensus_queue.cc:260] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.845919 23148 ts_tablet_manager.cc:1398] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 08:20:09.846436 23147 ts_tablet_manager.cc:1398] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:09.846698 23152 ts_tablet_manager.cc:1398] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:20:09.846822 23147 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:09.847079 23152 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:09.848767 23147 tablet_bootstrap.cc:654] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.848767 23152 tablet_bootstrap.cc:654] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:09.856721 23152 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:09.856877 23147 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:09.856930 23152 ts_tablet_manager.cc:1367] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240502 08:20:09.857101 23147 ts_tablet_manager.cc:1367] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 08:20:09.857841 23147 raft_consensus.cc:348] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.858027 23147 raft_consensus.cc:374] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.857910 23152 raft_consensus.cc:348] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.858101 23147 raft_consensus.cc:729] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.858134 23152 raft_consensus.cc:374] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:09.858218 23152 raft_consensus.cc:729] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:09.858340 23147 consensus_queue.cc:260] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.858495 23152 consensus_queue.cc:260] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:09.859069 23152 ts_tablet_manager.cc:1398] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:09.859745 23147 ts_tablet_manager.cc:1398] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:20:09.917101 23132 raft_consensus.cc:482] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:09.917311 23132 raft_consensus.cc:504] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.918040 23132 leader_election.cc:288] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:09.918745 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4862ea2f67c43a28689b464d8ec96e6" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:09.918813 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4862ea2f67c43a28689b464d8ec96e6" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:09.919075 21944 raft_consensus.cc:2454] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:09.919034 21552 raft_consensus.cc:2454] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:09.919618 21749 leader_election.cc:302] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:09.919939 23132 raft_consensus.cc:2781] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:09.920089 23132 raft_consensus.cc:482] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:09.920202 23132 raft_consensus.cc:3037] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:09.922561 23132 raft_consensus.cc:504] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.923125 23132 leader_election.cc:288] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:09.923655 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4862ea2f67c43a28689b464d8ec96e6" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:09.923766 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4862ea2f67c43a28689b464d8ec96e6" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:09.923895 21552 raft_consensus.cc:3037] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:09.924016 21944 raft_consensus.cc:3037] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:09.926095 21552 raft_consensus.cc:2454] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:09.926096 21944 raft_consensus.cc:2454] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:09.926643 21749 leader_election.cc:302] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:09.927018 23132 raft_consensus.cc:2781] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:09.927211 23132 raft_consensus.cc:686] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:09.927634 23132 consensus_queue.cc:237] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:09.931407 21404 catalog_manager.cc:5484] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:10.024168 23132 raft_consensus.cc:482] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:10.024392 23132 raft_consensus.cc:504] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.025111 23132 leader_election.cc:288] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:10.025633 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfc3eaed583458b8381d4e7b4d36aef" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:10.025758 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfc3eaed583458b8381d4e7b4d36aef" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:10.025892 21552 raft_consensus.cc:2454] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:10.026028 21683 raft_consensus.cc:2454] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:10.026384 21749 leader_election.cc:302] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:10.026733 23132 raft_consensus.cc:2781] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:10.026847 23132 raft_consensus.cc:482] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:10.026964 23132 raft_consensus.cc:3037] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.029516 23132 raft_consensus.cc:504] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.030294 23132 leader_election.cc:288] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:10.031111 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfc3eaed583458b8381d4e7b4d36aef" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:10.031380 21552 raft_consensus.cc:3037] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.031281 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfc3eaed583458b8381d4e7b4d36aef" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:10.031579 21683 raft_consensus.cc:3037] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.034718 21552 raft_consensus.cc:2454] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:10.035418 21749 leader_election.cc:302] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:10.035854 21683 raft_consensus.cc:2454] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:10.035997 23132 raft_consensus.cc:2781] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:10.036279 23132 raft_consensus.cc:686] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:10.036679 23132 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.040374 21404 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:10.173970 22885 raft_consensus.cc:482] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:10.174207 22885 raft_consensus.cc:504] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.174934 22885 leader_election.cc:288] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:10.175586 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efec2cc693bd4dd992d0bec22fca5419" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:10.175657 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efec2cc693bd4dd992d0bec22fca5419" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:10.175896 21944 raft_consensus.cc:2454] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:10.175900 21812 raft_consensus.cc:2454] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:10.176369 21619 leader_election.cc:302] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:10.176795 22885 raft_consensus.cc:2781] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:10.176909 22885 raft_consensus.cc:482] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:10.177039 22885 raft_consensus.cc:3037] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.179075 22885 raft_consensus.cc:504] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.179792 22885 leader_election.cc:288] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:10.180263 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efec2cc693bd4dd992d0bec22fca5419" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:10.180435 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efec2cc693bd4dd992d0bec22fca5419" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:10.180506 21944 raft_consensus.cc:3037] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.180677 21812 raft_consensus.cc:3037] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:10.182663 21944 raft_consensus.cc:2454] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:10.183202 21619 leader_election.cc:302] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:10.183426 21812 raft_consensus.cc:2454] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:10.183571 22885 raft_consensus.cc:2781] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:10.183760 22885 raft_consensus.cc:686] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:10.184068 22885 consensus_queue.cc:237] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:20:10.187153 21404 catalog_manager.cc:5484] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:20:10.390401 23132 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:10.399322 23135 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:10.467689 23135 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:10.475152 23135 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:10.666899 23040 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:10.668807 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:10.669313 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:20:10.672474 22885 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:10.681566 23163 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
W20240502 08:20:10.954831 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:36745, user_credentials={real_user=slave}} blocked reactor thread for 53464.3us
INSERT report
rows total: 2000
time total: 270.27 ms
time per row: 0.135135 ms
I20240502 08:20:11.118904 21407 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:11.129808 21662 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f6a0b2561419ca6bd832fd65e3196 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.130055 21531 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f6a0b2561419ca6bd832fd65e3196 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.130375 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f6a0b2561419ca6bd832fd65e3196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.130570 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f6a0b2561419ca6bd832fd65e3196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.130997 21661 tablet_service.cc:1467] Processing CreateTablet for tablet d62f6eec9ee04f93a3738a83f790292c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.131500 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62f6eec9ee04f93a3738a83f790292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.135452 21924 tablet_service.cc:1467] Processing CreateTablet for tablet d62f6eec9ee04f93a3738a83f790292c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.135721 21793 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f6a0b2561419ca6bd832fd65e3196 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.135998 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62f6eec9ee04f93a3738a83f790292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.136077 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f6a0b2561419ca6bd832fd65e3196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.137864 23224 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:11.140051 23224 tablet_bootstrap.cc:654] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.140556 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 203e7914da3f4494b239c6603513caee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.141045 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e7914da3f4494b239c6603513caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.141116 21923 tablet_service.cc:1467] Processing CreateTablet for tablet 203e7914da3f4494b239c6603513caee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.141546 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e7914da3f4494b239c6603513caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.141801 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 203e7914da3f4494b239c6603513caee (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.142369 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e7914da3f4494b239c6603513caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.148524 21531 tablet_service.cc:1467] Processing CreateTablet for tablet d62f6eec9ee04f93a3738a83f790292c (DEFAULT_TABLE table=rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:11.149137 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62f6eec9ee04f93a3738a83f790292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:11.163322 23227 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:11.166713 23228 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:11.170403 23228 tablet_bootstrap.cc:654] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.172710 23227 tablet_bootstrap.cc:654] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.173437 23226 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:11.175881 23227 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:11.176136 23227 ts_tablet_manager.cc:1367] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:20:11.176584 23226 tablet_bootstrap.cc:654] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.176968 23227 raft_consensus.cc:348] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.177193 23227 raft_consensus.cc:374] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.177285 23227 raft_consensus.cc:729] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.177506 23224 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:11.177716 23224 ts_tablet_manager.cc:1367] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.019s
I20240502 08:20:11.177665 23227 consensus_queue.cc:260] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.178300 23227 ts_tablet_manager.cc:1398] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:11.178645 23227 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:11.178566 23224 raft_consensus.cc:348] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.178779 23224 raft_consensus.cc:374] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.178855 23224 raft_consensus.cc:729] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.179083 23224 consensus_queue.cc:260] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.180526 23227 tablet_bootstrap.cc:654] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.181707 23228 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:11.181828 23224 ts_tablet_manager.cc:1398] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:20:11.181949 23228 ts_tablet_manager.cc:1367] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 08:20:11.182224 23224 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:11.182971 23228 raft_consensus.cc:348] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.183302 23228 raft_consensus.cc:374] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.183463 23228 raft_consensus.cc:729] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.183838 23228 consensus_queue.cc:260] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.183990 23224 tablet_bootstrap.cc:654] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.184720 23228 ts_tablet_manager.cc:1398] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:11.185029 23228 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:11.186517 23228 tablet_bootstrap.cc:654] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.189308 23226 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:11.189577 23226 ts_tablet_manager.cc:1367] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:20:11.190908 23226 raft_consensus.cc:348] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.191228 23226 raft_consensus.cc:374] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.191332 23227 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:11.191576 23227 ts_tablet_manager.cc:1367] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:20:11.191586 23226 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.191843 23226 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.192435 23226 ts_tablet_manager.cc:1398] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:20:11.192334 23227 raft_consensus.cc:348] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.192557 23227 raft_consensus.cc:374] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.192704 23227 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.192893 23226 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:11.193097 23227 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.193897 23227 ts_tablet_manager.cc:1398] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:11.194584 23226 tablet_bootstrap.cc:654] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.195935 23228 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:11.196195 23228 ts_tablet_manager.cc:1367] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 08:20:11.197049 23228 raft_consensus.cc:348] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.197391 23228 raft_consensus.cc:374] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.197510 23228 raft_consensus.cc:729] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.197853 23228 consensus_queue.cc:260] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.198690 23228 ts_tablet_manager.cc:1398] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:11.215590 23226 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:11.215822 23226 ts_tablet_manager.cc:1367] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.002s
I20240502 08:20:11.216508 23226 raft_consensus.cc:348] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.216715 23226 raft_consensus.cc:374] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.216910 23226 raft_consensus.cc:729] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.217296 23226 consensus_queue.cc:260] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.217978 23226 ts_tablet_manager.cc:1398] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:20:11.218051 23224 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:11.218253 23224 ts_tablet_manager.cc:1367] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.008s
I20240502 08:20:11.219151 23224 raft_consensus.cc:348] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.219419 23224 raft_consensus.cc:374] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.219501 23224 raft_consensus.cc:729] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.219756 23224 consensus_queue.cc:260] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.220332 23224 ts_tablet_manager.cc:1398] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:11.220871 23214 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:11.221042 23214 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.221863 23214 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:11.222530 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:11.222709 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:11.223001 21943 raft_consensus.cc:2454] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
W20240502 08:20:11.223330 21749 leader_election.cc:341] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:20:11.223661 21749 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters: cfff665685a14d8796aece26addab59c
I20240502 08:20:11.224855 23214 raft_consensus.cc:2781] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:11.224982 23214 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:11.225075 23214 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.220738 23224 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:11.228070 23214 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.228724 23214 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:11.229497 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:11.229801 21943 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.229739 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
W20240502 08:20:11.230796 21749 leader_election.cc:341] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:20:11.232930 21943 raft_consensus.cc:2454] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:11.234578 21749 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8; no voters: cfff665685a14d8796aece26addab59c
I20240502 08:20:11.235018 23224 tablet_bootstrap.cc:654] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:11.235198 23214 raft_consensus.cc:2781] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:11.235785 23214 raft_consensus.cc:686] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:11.236272 23214 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.241267 23224 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:11.241487 23224 ts_tablet_manager.cc:1367] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.007s
I20240502 08:20:11.242183 23224 raft_consensus.cc:348] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.242429 23224 raft_consensus.cc:374] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:11.242308 21407 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:11.242528 23224 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:11.243499 23224 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.244228 23224 ts_tablet_manager.cc:1398] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:20:11.361393 22918 raft_consensus.cc:482] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:11.361681 22918 raft_consensus.cc:504] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.362428 22918 leader_election.cc:288] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:11.362977 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62f6eec9ee04f93a3738a83f790292c" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:11.363330 21683 raft_consensus.cc:2454] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:11.363334 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62f6eec9ee04f93a3738a83f790292c" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:11.363552 21943 raft_consensus.cc:2454] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 0.
I20240502 08:20:11.364012 21488 leader_election.cc:302] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:11.364305 22918 raft_consensus.cc:2781] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:11.364414 22918 raft_consensus.cc:482] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:11.364532 22918 raft_consensus.cc:3037] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.366796 22918 raft_consensus.cc:504] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.367470 22918 leader_election.cc:288] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:11.368016 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62f6eec9ee04f93a3738a83f790292c" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:11.368074 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62f6eec9ee04f93a3738a83f790292c" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:11.368268 21943 raft_consensus.cc:3037] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.368285 21683 raft_consensus.cc:3037] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.371275 21683 raft_consensus.cc:2454] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:20:11.371295 21943 raft_consensus.cc:2454] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 1.
I20240502 08:20:11.371747 21488 leader_election.cc:302] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:11.372067 22918 raft_consensus.cc:2781] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:11.372269 22918 raft_consensus.cc:686] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Becoming Leader. State: Replica: cfff665685a14d8796aece26addab59c, State: Running, Role: LEADER
I20240502 08:20:11.372547 22918 consensus_queue.cc:237] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:11.375811 21411 catalog_manager.cc:5484] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c reported cstate change: term changed from 0 to 1, leader changed from <none> to cfff665685a14d8796aece26addab59c (127.8.68.193). New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:20:11.395215 23214 raft_consensus.cc:482] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:11.395514 23214 raft_consensus.cc:504] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.396230 23214 leader_election.cc:288] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:11.396833 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f6a0b2561419ca6bd832fd65e3196" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:11.397009 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f6a0b2561419ca6bd832fd65e3196" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:11.397102 21683 raft_consensus.cc:2454] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:11.397280 21551 raft_consensus.cc:2454] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:11.397552 21749 leader_election.cc:302] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:11.397900 23214 raft_consensus.cc:2781] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:11.398017 23214 raft_consensus.cc:482] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:11.398162 23214 raft_consensus.cc:3037] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.400281 23214 raft_consensus.cc:504] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.400894 23214 leader_election.cc:288] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:11.401460 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f6a0b2561419ca6bd832fd65e3196" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:11.401481 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f6a0b2561419ca6bd832fd65e3196" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:11.401669 21551 raft_consensus.cc:3037] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.401721 21683 raft_consensus.cc:3037] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.404088 21551 raft_consensus.cc:2454] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:11.404089 21683 raft_consensus.cc:2454] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:11.404551 21749 leader_election.cc:302] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:11.404928 23214 raft_consensus.cc:2781] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:11.405143 23214 raft_consensus.cc:686] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:11.405444 23214 consensus_queue.cc:237] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.408969 21411 catalog_manager.cc:5484] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:11.484714 22918 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:11.484968 22918 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:11.485893 22918 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:11.486706 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:11.486704 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:11.487020 21943 raft_consensus.cc:2379] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cfff665685a14d8796aece26addab59c in current term 1: Already voted for candidate 13e764b27ed449c09aee719cdb2597f8 in this term.
I20240502 08:20:11.487609 21488 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8
I20240502 08:20:11.487918 22918 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:11.490368 22918 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:20:11.653326 23214 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:11.659134 23214 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:11.829284 22918 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:11.835829 23248 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:11.870857 23214 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:11.871188 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:11.871695 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:20:11.876533 23214 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:11.887634 23241 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 273.698 ms
time per row: 0.136849 ms
I20240502 08:20:12.325366 21406 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59060:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:20:12.335887 21662 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45b1da348e4bb89f021df103a0603f (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.335996 21924 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45b1da348e4bb89f021df103a0603f (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.336427 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45b1da348e4bb89f021df103a0603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.336436 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45b1da348e4bb89f021df103a0603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.338510 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 12c17796e5d04014aafdac46984f4ba9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.339010 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.340354 21661 tablet_service.cc:1467] Processing CreateTablet for tablet d8042f6ab6404c5a9f8cce0b8f971542 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.340433 21793 tablet_service.cc:1467] Processing CreateTablet for tablet d8042f6ab6404c5a9f8cce0b8f971542 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.340543 21792 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45b1da348e4bb89f021df103a0603f (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.340807 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8042f6ab6404c5a9f8cce0b8f971542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.340847 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8042f6ab6404c5a9f8cce0b8f971542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.341068 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45b1da348e4bb89f021df103a0603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.344022 21923 tablet_service.cc:1467] Processing CreateTablet for tablet d8042f6ab6404c5a9f8cce0b8f971542 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.343971 21663 tablet_service.cc:1467] Processing CreateTablet for tablet 12c17796e5d04014aafdac46984f4ba9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.344049 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 12c17796e5d04014aafdac46984f4ba9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:20:12.344513 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8042f6ab6404c5a9f8cce0b8f971542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.345700 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.345542 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:12.359294 23308 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:12.361842 23308 tablet_bootstrap.cc:654] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.368624 23308 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:12.368803 23308 ts_tablet_manager.cc:1367] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 08:20:12.369532 23308 raft_consensus.cc:348] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.369925 23308 raft_consensus.cc:374] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.370234 23308 raft_consensus.cc:729] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.370539 23308 consensus_queue.cc:260] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.371393 23308 ts_tablet_manager.cc:1398] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:12.371718 23308 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:12.373641 23308 tablet_bootstrap.cc:654] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.386446 23307 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:12.385560 23310 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:12.386667 23312 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:20:12.387769 23308 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b: No bootstrap required, opened a new log
I20240502 08:20:12.387961 23308 ts_tablet_manager.cc:1367] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20240502 08:20:12.388655 23310 tablet_bootstrap.cc:654] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.388672 23308 raft_consensus.cc:348] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.388890 23308 raft_consensus.cc:374] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.388968 23308 raft_consensus.cc:729] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.389008 23307 tablet_bootstrap.cc:654] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.389226 23308 consensus_queue.cc:260] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.389868 23308 ts_tablet_manager.cc:1398] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:12.391624 23312 tablet_bootstrap.cc:654] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.397348 23312 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c: No bootstrap required, opened a new log
I20240502 08:20:12.397531 23312 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:20:12.398401 23312 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.398656 23312 raft_consensus.cc:374] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.398772 23312 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.398943 23310 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:12.399103 23310 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240502 08:20:12.399026 23312 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.399670 23312 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:20:12.400146 23310 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.400362 23310 raft_consensus.cc:374] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.400444 23310 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.400658 23310 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.401378 23310 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:20:12.402971 23307 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:12.403214 23307 ts_tablet_manager.cc:1367] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20240502 08:20:12.403455 23310 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:12.404394 23307 raft_consensus.cc:348] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.404707 23307 raft_consensus.cc:374] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.404865 23307 raft_consensus.cc:729] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.405405 23310 tablet_bootstrap.cc:654] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.405309 23307 consensus_queue.cc:260] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.406006 23307 ts_tablet_manager.cc:1398] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:12.406364 23307 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:12.408151 23307 tablet_bootstrap.cc:654] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.408360 23310 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:12.408543 23310 ts_tablet_manager.cc:1367] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 08:20:12.409344 23310 raft_consensus.cc:348] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.409538 23310 raft_consensus.cc:374] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.409619 23310 raft_consensus.cc:729] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.409866 23310 consensus_queue.cc:260] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.410487 23310 ts_tablet_manager.cc:1398] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:20:12.410781 23310 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:20:12.412356 23307 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:12.412460 23310 tablet_bootstrap.cc:654] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.412484 23307 ts_tablet_manager.cc:1367] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240502 08:20:12.413194 23307 raft_consensus.cc:348] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.413398 23307 raft_consensus.cc:374] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.413478 23307 raft_consensus.cc:729] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.413758 23307 consensus_queue.cc:260] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.414251 23307 ts_tablet_manager.cc:1398] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:20:12.414543 23307 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:20:12.414651 23310 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: No bootstrap required, opened a new log
I20240502 08:20:12.414840 23310 ts_tablet_manager.cc:1367] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:20:12.415622 23310 raft_consensus.cc:348] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.415928 23310 raft_consensus.cc:374] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.416052 23307 tablet_bootstrap.cc:654] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Neither blocks nor log segments found. Creating new log.
I20240502 08:20:12.416040 23310 raft_consensus.cc:729] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.416298 23310 consensus_queue.cc:260] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.416934 23310 ts_tablet_manager.cc:1398] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:12.417918 23307 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: No bootstrap required, opened a new log
I20240502 08:20:12.418066 23307 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:20:12.418759 23307 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.418936 23307 raft_consensus.cc:374] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:20:12.419044 23307 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:20:12.419297 23307 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.419827 23307 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:12.467725 23214 raft_consensus.cc:482] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:12.467945 23214 raft_consensus.cc:504] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.468750 23214 leader_election.cc:288] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:12.469342 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8042f6ab6404c5a9f8cce0b8f971542" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238" is_pre_election: true
I20240502 08:20:12.469379 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8042f6ab6404c5a9f8cce0b8f971542" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:12.469604 21683 raft_consensus.cc:2454] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:12.469631 21943 raft_consensus.cc:2454] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 0.
I20240502 08:20:12.470103 21749 leader_election.cc:302] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:12.470402 23214 raft_consensus.cc:2781] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:12.470521 23214 raft_consensus.cc:482] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:12.470624 23214 raft_consensus.cc:3037] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.472720 23214 raft_consensus.cc:504] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.473323 23214 leader_election.cc:288] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:12.473953 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8042f6ab6404c5a9f8cce0b8f971542" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:12.474016 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8042f6ab6404c5a9f8cce0b8f971542" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:12.474201 21683 raft_consensus.cc:3037] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.474226 21943 raft_consensus.cc:3037] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.476398 21943 raft_consensus.cc:2454] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:12.476420 21683 raft_consensus.cc:2454] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 1.
I20240502 08:20:12.476879 21749 leader_election.cc:302] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:12.477226 23255 raft_consensus.cc:2781] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:12.477382 23255 raft_consensus.cc:686] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:12.477741 23255 consensus_queue.cc:237] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:12.481297 21408 catalog_manager.cc:5484] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:12.485862 23290 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:12.486034 23290 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.486760 23290 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:12.487334 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:12.487375 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
I20240502 08:20:12.487583 21812 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:12.487598 21552 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:12.488036 21618 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:12.488334 23290 raft_consensus.cc:2781] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:12.488444 23290 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:12.488530 23290 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.490483 23290 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.491050 23290 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:12.491604 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:12.491860 21812 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.493922 21812 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:12.494354 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:12.494572 21552 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.496838 21552 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:12.497359 21618 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238, cfff665685a14d8796aece26addab59c; no voters:
I20240502 08:20:12.497777 23290 raft_consensus.cc:2781] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:12.497934 23290 raft_consensus.cc:686] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:12.498224 23290 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:20:12.501346 21408 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:20:12.539817 22885 raft_consensus.cc:482] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:20:12.540041 22885 raft_consensus.cc:504] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.540818 22885 leader_election.cc:288] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:12.541440 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45b1da348e4bb89f021df103a0603f" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:20:12.541450 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45b1da348e4bb89f021df103a0603f" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b" is_pre_election: true
I20240502 08:20:12.541666 21812 raft_consensus.cc:2454] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:12.541724 21943 raft_consensus.cc:2454] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 0.
I20240502 08:20:12.542032 21618 leader_election.cc:302] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:12.542327 22885 raft_consensus.cc:2781] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:20:12.542419 22885 raft_consensus.cc:482] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:20:12.542533 22885 raft_consensus.cc:3037] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.544503 22885 raft_consensus.cc:504] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.545135 22885 leader_election.cc:288] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Requested vote from peers 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:12.545682 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45b1da348e4bb89f021df103a0603f" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:12.545928 21812 raft_consensus.cc:3037] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.545936 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45b1da348e4bb89f021df103a0603f" candidate_uuid: "60677121510e47a5b51d1eb22f9aa238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:12.546164 21943 raft_consensus.cc:3037] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:20:12.548956 21812 raft_consensus.cc:2454] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:12.548995 21943 raft_consensus.cc:2454] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60677121510e47a5b51d1eb22f9aa238 in term 1.
I20240502 08:20:12.549528 21619 leader_election.cc:302] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:12.549839 23290 raft_consensus.cc:2781] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:20:12.550010 23290 raft_consensus.cc:686] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Leader. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:12.550323 23290 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:20:12.553658 21408 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:20:12.906102 23290 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:12.914326 23290 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:20:13.009195 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:13.009601 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:13.017447 23323 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:20:13.033779 23214 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:13.050168 23255 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 08:20:13.077653 23290 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:20:13.083032 22885 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 282.48 ms
time per row: 0.14124 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:20:14.452627 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:45373
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:44613
--webserver_port=40675
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:20:14.836972 23388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:14.837320 23388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:20:14.837533 23388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:14.845947 23388 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:20:14.846091 23388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:20:14.846215 23388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240502 08:20:14.856084 23388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:45373
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=40675
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:20:14.857925 23388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:20:14.859805 23388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:20:14.873839 23395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:14.874370 23397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:14.874641 23394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:20:14.875494 23388 server_base.cc:1034] running on GCE node
I20240502 08:20:14.876168 23388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:20:14.877219 23388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:20:14.878470 23388 hybrid_clock.cc:648] HybridClock initialized: now 1714638014878424 us; error 35 us; skew 500 ppm
I20240502 08:20:14.878865 23388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:20:14.881546 23388 webserver.cc:460] Webserver started at http://127.8.68.197:40675/ using document root <none> and password file <none>
I20240502 08:20:14.882387 23388 fs_manager.cc:359] Metadata directory not provided
I20240502 08:20:14.882531 23388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:20:14.888895 23388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:20:14.892469 23403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:14.894127 23388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:20:14.894315 23388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "20b637fbec5444dd91b3af9cdda482f5"
format_stamp: "Formatted at 2024-05-02 08:19:53 on dist-test-slave-2dhs"
I20240502 08:20:14.895598 23388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:20:14.925691 23388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:20:14.926559 23388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:20:14.926836 23388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:20:14.927907 23388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:20:14.929574 23388 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:20:14.929656 23388 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:14.929766 23388 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:20:14.929808 23388 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:14.964962 23388 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:45373
I20240502 08:20:14.965016 23515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:45373 every 8 connection(s)
I20240502 08:20:14.966552 23388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:20:14.973521 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 23388
I20240502 08:20:14.974340 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:41299
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:44613
--webserver_port=39265
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:20:14.982792 23516 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:20:14.983150 23516 heartbeater.cc:464] Registering TS with master...
I20240502 08:20:14.984009 23516 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:20:14.985785 21409 ts_manager.cc:194] Re-registered known tserver with Master: 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:20:14.986958 21409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:58527
W20240502 08:20:15.375289 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:15.375680 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:20:15.375783 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:15.383638 23520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:20:15.383783 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:20:15.383911 23520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240502 08:20:15.395284 23520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:41299
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=39265
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:20:15.397010 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:20:15.398960 23520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:20:15.412257 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:15.412222 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:15.413041 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:20:15.413654 23520 server_base.cc:1034] running on GCE node
I20240502 08:20:15.414463 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:20:15.415715 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:20:15.416970 23520 hybrid_clock.cc:648] HybridClock initialized: now 1714638015416906 us; error 45 us; skew 500 ppm
I20240502 08:20:15.417371 23520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:20:15.420228 23520 webserver.cc:460] Webserver started at http://127.8.68.198:39265/ using document root <none> and password file <none>
I20240502 08:20:15.421181 23520 fs_manager.cc:359] Metadata directory not provided
I20240502 08:20:15.421336 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:20:15.427523 23520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:20:15.430761 23534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.432499 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:20:15.432704 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b9bfd2541144dad86507451343640d3"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:20:15.433827 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:20:15.459882 23520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:20:15.460776 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:20:15.461063 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:20:15.462083 23520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:20:15.463953 23520 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:20:15.464053 23520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.464190 23520 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:20:15.464257 23520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.500655 23520 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:41299
I20240502 08:20:15.500787 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:41299 every 8 connection(s)
I20240502 08:20:15.502425 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:20:15.507076 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 23520
I20240502 08:20:15.507843 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:33397
--local_ip_for_outbound_sockets=127.8.68.199
--tserver_master_addrs=127.8.68.254:44613
--webserver_port=36495
--webserver_interface=127.8.68.199
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:20:15.518903 23648 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:20:15.519299 23648 heartbeater.cc:464] Registering TS with master...
I20240502 08:20:15.520468 23648 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:20:15.522428 21409 ts_manager.cc:194] Re-registered known tserver with Master: 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299)
I20240502 08:20:15.523660 21409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:38565
W20240502 08:20:15.903190 23652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:15.903594 23652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:20:15.903689 23652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:15.912019 23652 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:20:15.912163 23652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:20:15.912286 23652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240502 08:20:15.922318 23652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:33397
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=36495
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:20:15.923907 23652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:20:15.926012 23652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:20:15.940681 23659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:15.941260 23658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:15.941862 23661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:20:15.942322 23652 server_base.cc:1034] running on GCE node
I20240502 08:20:15.943025 23652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:20:15.944172 23652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:20:15.945441 23652 hybrid_clock.cc:648] HybridClock initialized: now 1714638015945375 us; error 60 us; skew 500 ppm
I20240502 08:20:15.945827 23652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:20:15.948753 23652 webserver.cc:460] Webserver started at http://127.8.68.199:36495/ using document root <none> and password file <none>
I20240502 08:20:15.949708 23652 fs_manager.cc:359] Metadata directory not provided
I20240502 08:20:15.949889 23652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:20:15.956789 23652 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 08:20:15.960067 23667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.961705 23652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:20:15.961934 23652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
format_stamp: "Formatted at 2024-05-02 08:19:54 on dist-test-slave-2dhs"
I20240502 08:20:15.962853 23652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:20:15.979214 23652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:20:15.980243 23652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:20:15.980562 23652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:20:15.981626 23652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:20:15.983423 23652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:20:15.983532 23652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.983673 23652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:20:15.983728 23652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:15.993906 23516 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:20:16.022579 23652 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:33397
I20240502 08:20:16.022622 23781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:33397 every 8 connection(s)
I20240502 08:20:16.024616 23652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:20:16.031301 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 23652
I20240502 08:20:16.032191 8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskivhcCF/build/asan/bin/kudu
/tmp/dist-test-taskivhcCF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:40881
--local_ip_for_outbound_sockets=127.8.68.200
--tserver_master_addrs=127.8.68.254:44613
--webserver_port=44633
--webserver_interface=127.8.68.200
--builtin_ntp_servers=127.8.68.212:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:20:16.051717 23782 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:20:16.052124 23782 heartbeater.cc:464] Registering TS with master...
I20240502 08:20:16.053222 23782 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:20:16.054942 21409 ts_manager.cc:194] Re-registered known tserver with Master: fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397)
I20240502 08:20:16.056033 21409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:37877
W20240502 08:20:16.429944 23786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:16.430316 23786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:20:16.430480 23786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:16.438863 23786 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:20:16.439009 23786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:20:16.439162 23786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240502 08:20:16.449165 23786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:40881
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=44633
--tserver_master_addrs=127.8.68.254:44613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:20:16.450765 23786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:20:16.453301 23786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:20:16.466874 23792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:16.467502 23795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:20:16.468101 23793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:20:16.468675 23786 server_base.cc:1034] running on GCE node
I20240502 08:20:16.469337 23786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:20:16.470362 23786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:20:16.471588 23786 hybrid_clock.cc:648] HybridClock initialized: now 1714638016471532 us; error 46 us; skew 500 ppm
I20240502 08:20:16.471980 23786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:20:16.474877 23786 webserver.cc:460] Webserver started at http://127.8.68.200:44633/ using document root <none> and password file <none>
I20240502 08:20:16.476008 23786 fs_manager.cc:359] Metadata directory not provided
I20240502 08:20:16.476161 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:20:16.482954 23786 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:20:16.486414 23802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:16.488229 23786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:20:16.488459 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "95f32202b9db457fb09872f6f51ea8ba"
format_stamp: "Formatted at 2024-05-02 08:19:55 on dist-test-slave-2dhs"
I20240502 08:20:16.489667 23786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:20:16.509171 23786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:20:16.510282 23786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:20:16.510663 23786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:20:16.511876 23786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:20:16.513489 23786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:20:16.513571 23786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:16.513676 23786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:20:16.513721 23786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:20:16.527405 23648 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:20:16.550747 23786 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:40881
I20240502 08:20:16.550897 23915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:40881 every 8 connection(s)
I20240502 08:20:16.552307 23786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:20:16.560606 8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskivhcCF/build/asan/bin/kudu as pid 23786
I20240502 08:20:16.568724 23916 heartbeater.cc:347] Connected to a master server at 127.8.68.254:44613
I20240502 08:20:16.569114 23916 heartbeater.cc:464] Registering TS with master...
I20240502 08:20:16.570170 23916 heartbeater.cc:511] Master 127.8.68.254:44613 requested a full tablet report, sending...
I20240502 08:20:16.571779 21409 ts_manager.cc:194] Re-registered known tserver with Master: 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
I20240502 08:20:16.572607 21409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:53653
I20240502 08:20:17.059279 23782 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
I20240502 08:20:17.576205 23916 heartbeater.cc:503] Master 127.8.68.254:44613 was elected leader, sending a full tablet report...
W20240502 08:20:17.954142 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 45580.3us
I20240502 08:20:18.094560 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.190821 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:18.209630 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:18.213999 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.218901 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.279644 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.311654 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.333328 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.361943 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:18.373786 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.386329 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.421576 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.430135 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.433615 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.456600 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:18.472357 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.477069 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.490200 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:18.494480 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.524749 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.575807 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:18.655063 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.674351 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.706683 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.918684 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.918989 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.923624 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:18.928515 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.935976 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.944578 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.949700 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:18.959960 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:18.960855 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:18.976192 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:18.996977 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:19.008510 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:19.061661 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:19.103737 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:19.130434 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:19.142589 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:19.999761 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.000892 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.012115 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.016129 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:20.081315 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.091344 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.147920 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.156836 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.256340 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.270121 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:20.279541 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:20.283741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.289980 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.314052 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.352051 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:20.363021 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:20.390831 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.391049 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.406598 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.412279 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.446916 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:20.472311 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.498760 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.524556 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.619786 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.662681 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:20.704180 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.708112 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.793649 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:20.839362 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.890592 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.895372 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:20.913347 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:20.945744 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:20.999351 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:21.006654 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:21.015062 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:21.043207 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:21.063828 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:21.080626 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:21.576871 21487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 60677.6us
I20240502 08:20:21.753178 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:21.892513 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:21.902448 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:21.930486 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:21.933106 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:21.988973 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.004244 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:22.052583 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.052660 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:22.075415 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.083797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.092048 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.105324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.112953 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:20:22.118466 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 85716us
I20240502 08:20:22.120992 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.154528 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:22.155385 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:22.194509 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.233647 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.234087 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.236356 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.266974 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.297402 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:22.303352 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.328943 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.332309 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.358783 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.363588 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:22.401881 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:22.406670 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.431031 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.436632 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.559952 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.601567 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:22.602103 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.611701 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.660550 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:22.670014 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:22.725174 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:22.728775 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:23.019539 21683 consensus_queue.cc:237] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:23.048883 21551 raft_consensus.cc:1261] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.052013 24628 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:23.080286 21813 raft_consensus.cc:1261] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.085013 24628 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:23.106014 23930 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:23.108649 21813 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:23.116010 21403 catalog_manager.cc:5484] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:20:23.121747 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3898bfdb9c3d4a05af78f79fd2de61e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:23.127681 21550 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:23.356927 21813 consensus_queue.cc:237] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:23.383353 21683 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.384446 24652 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:23.415088 21550 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.415933 24652 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:23.451431 21746 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf79c69fd614284ba45fd052cac7ebf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:23.453357 24804 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:23.455580 21550 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:23.463263 21407 catalog_manager.cc:5484] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:23.471880 21683 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:23.502480 24811 ts_tablet_manager.cc:902] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:23.551167 24811 tablet_copy_client.cc:308] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:23.622598 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 3898bfdb9c3d4a05af78f79fd2de61e2 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:20:23.622960 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 3898bfdb9c3d4a05af78f79fd2de61e2 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-3898bfdb9c3d4a05af78f79fd2de61e2
I20240502 08:20:23.627732 21703 tablet_copy_source_session.cc:205] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:23.634575 24811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3898bfdb9c3d4a05af78f79fd2de61e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:23.732228 24811 tablet_copy_client.cc:783] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:23.732699 24811 tablet_copy_client.cc:647] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:23.739490 24811 tablet_copy_client.cc:523] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:23.744904 21812 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:23.746232 24811 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:23.753154 21943 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.754010 24804 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:23.755775 21550 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:23.756510 24804 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:23.771314 24811 log.cc:824] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Log is configured to *not* fsync() on all Append() calls
W20240502 08:20:23.772259 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 852613f14b324b38b5eecc29d4c602cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:23.778182 24652 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:23.783703 21943 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:23.804862 21550 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:23.817858 21402 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:23.845206 24811 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:23.850196 24811 tablet_bootstrap.cc:492] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:23.851698 24811 ts_tablet_manager.cc:1367] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.106s user 0.074s sys 0.004s
I20240502 08:20:23.860531 24811 raft_consensus.cc:348] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:23.868189 24811 raft_consensus.cc:729] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:23.869984 24811 consensus_queue.cc:260] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:23.886741 24811 ts_tablet_manager.cc:1398] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.034s user 0.009s sys 0.004s
I20240502 08:20:23.892738 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-3898bfdb9c3d4a05af78f79fd2de61e2 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:20:23.893038 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-3898bfdb9c3d4a05af78f79fd2de61e2 on tablet 3898bfdb9c3d4a05af78f79fd2de61e2 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:23.939441 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:23.929800 24895 ts_tablet_manager.cc:902] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:23.966202 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:23.974223 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.012534 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:24.056881 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:24.074560 23471 raft_consensus.cc:1203] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:24.110637 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.150530 24895 tablet_copy_client.cc:308] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:24.175545 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:24.216212 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.221884 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.232088 24962 ts_tablet_manager.cc:902] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:24.238416 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.242290 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet ebf79c69fd614284ba45fd052cac7ebf from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:35393)
I20240502 08:20:24.242645 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet ebf79c69fd614284ba45fd052cac7ebf from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:35393: session id = 95f32202b9db457fb09872f6f51ea8ba-ebf79c69fd614284ba45fd052cac7ebf
I20240502 08:20:24.243417 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.246518 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.246843 21833 tablet_copy_source_session.cc:205] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:24.273653 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.280810 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf79c69fd614284ba45fd052cac7ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:24.283677 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:24.307317 24895 tablet_copy_client.cc:783] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:24.307749 24895 tablet_copy_client.cc:647] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:24.310870 24895 tablet_copy_client.cc:523] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:24.314621 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.316092 24895 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:24.341324 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.360822 24895 log.cc:824] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Log is configured to *not* fsync() on all Append() calls
I20240502 08:20:24.366209 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.381103 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.423965 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.426463 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.444473 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:24.456761 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.460919 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:24.462903 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.574883 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:24.609246 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.615573 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.643143 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:24.645442 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.664475 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:24.670571 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.679600 24962 tablet_copy_client.cc:308] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:24.681787 24895 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:24.682474 24895 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:24.683053 24895 ts_tablet_manager.cc:1367] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.367s user 0.064s sys 0.010s
I20240502 08:20:24.725000 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:24.774768 24895 raft_consensus.cc:348] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:24.777892 24895 raft_consensus.cc:729] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:24.778977 24895 consensus_queue.cc:260] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:24.801903 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 852613f14b324b38b5eecc29d4c602cd from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:47771)
I20240502 08:20:24.802145 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 852613f14b324b38b5eecc29d4c602cd from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:47771: session id = 4b9bfd2541144dad86507451343640d3-852613f14b324b38b5eecc29d4c602cd
I20240502 08:20:24.805514 21833 tablet_copy_source_session.cc:205] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:24.812634 24962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852613f14b324b38b5eecc29d4c602cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:24.855859 24895 ts_tablet_manager.cc:1398] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.173s user 0.009s sys 0.006s
I20240502 08:20:24.857194 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-ebf79c69fd614284ba45fd052cac7ebf received from {username='slave'} at 127.8.68.200:35393
I20240502 08:20:24.857404 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-ebf79c69fd614284ba45fd052cac7ebf on tablet ebf79c69fd614284ba45fd052cac7ebf with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:24.875660 21812 consensus_queue.cc:237] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:24.914426 24962 tablet_copy_client.cc:783] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:24.915037 24962 tablet_copy_client.cc:647] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:24.922938 24962 tablet_copy_client.cc:523] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:24.938620 24962 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:24.947824 21683 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:24.948637 24652 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:24.961279 21551 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:24.963953 24652 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:24.985769 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b3f6a0b2561419ca6bd832fd65e3196. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:24.995584 24962 log.cc:824] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:20:24.997355 25012 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:24.999917 21551 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.012253 21403 catalog_manager.cc:5484] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.094838 21683 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.102262 23871 raft_consensus.cc:1203] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:25.105468 24962 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:25.106159 24962 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:25.106719 24962 ts_tablet_manager.cc:1367] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.168s user 0.066s sys 0.007s
I20240502 08:20:25.134903 24962 raft_consensus.cc:348] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:25.135566 24962 raft_consensus.cc:729] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:25.136436 24962 consensus_queue.cc:260] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:25.155472 24962 ts_tablet_manager.cc:1398] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.049s user 0.016s sys 0.000s
I20240502 08:20:25.157105 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-852613f14b324b38b5eecc29d4c602cd received from {username='slave'} at 127.8.68.198:47771
I20240502 08:20:25.157297 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 4b9bfd2541144dad86507451343640d3-852613f14b324b38b5eecc29d4c602cd on tablet 852613f14b324b38b5eecc29d4c602cd with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:25.227687 21683 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:25.244051 21551 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.245023 25034 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.256049 21812 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.259316 24990 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:25.270478 21618 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12c17796e5d04014aafdac46984f4ba9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.281229 21683 consensus_queue.cc:237] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:25.301322 21944 raft_consensus.cc:1261] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.302726 25034 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:25.305711 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efec2cc693bd4dd992d0bec22fca5419. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.316198 21812 raft_consensus.cc:1261] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.317065 25034 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.323617 24990 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.330255 21550 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.338263 21406 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.339299 21812 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:25.363168 25034 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.366920 21813 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.370718 21944 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.373382 21407 catalog_manager.cc:5484] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.386785 25100 ts_tablet_manager.cc:902] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:25.406162 21683 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:25.414335 21944 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.415829 24990 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.440639 21943 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:25.445169 21551 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.446348 21683 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.447484 24694 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.451347 24694 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:25.451519 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182a6f2df91242cfa10f05ef788a19cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.460183 23603 raft_consensus.cc:1203] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:25.472082 21550 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.473817 25034 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.480662 25100 tablet_copy_client.cc:308] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
W20240502 08:20:25.499624 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76bb1e3e2b9d4e31bf93ac1ce08a7833. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.510145 25034 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.523799 21943 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.533885 21550 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.535022 21406 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.546583 25149 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.550971 21683 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.558431 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 7b3f6a0b2561419ca6bd832fd65e3196 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:43557)
I20240502 08:20:25.558650 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 7b3f6a0b2561419ca6bd832fd65e3196 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:43557: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-7b3f6a0b2561419ca6bd832fd65e3196
I20240502 08:20:25.561728 21833 tablet_copy_source_session.cc:205] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:25.569518 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f6a0b2561419ca6bd832fd65e3196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:25.573163 21550 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.585947 21406 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:25.616390 25100 tablet_copy_client.cc:783] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:25.616977 25100 tablet_copy_client.cc:647] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:25.620136 25100 tablet_copy_client.cc:523] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:25.626857 25100 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:25.650296 21550 consensus_queue.cc:237] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:25.663316 21552 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:25.682351 21943 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.683599 24526 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.688210 21682 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.689738 24526 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.709122 25100 log.cc:824] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Log is configured to *not* fsync() on all Append() calls
I20240502 08:20:25.716979 21683 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.717936 24526 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.730176 21943 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.731014 24526 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.737205 25202 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:25.738024 21943 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:25.744901 21407 catalog_manager.cc:5484] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:25.756059 21683 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
W20240502 08:20:25.760380 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d62f6eec9ee04f93a3738a83f790292c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.760448 25202 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.763892 21683 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:25.777258 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c0a886b3ba04737a6988c3b58c9698e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.775043 21943 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.780323 21403 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:25.811133 25209 ts_tablet_manager.cc:902] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:25.834059 25100 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:25.834888 25100 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:25.838898 25100 ts_tablet_manager.cc:1367] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.212s user 0.076s sys 0.005s
I20240502 08:20:25.851068 25228 ts_tablet_manager.cc:902] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:25.867136 25100 raft_consensus.cc:348] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:25.870535 25100 raft_consensus.cc:729] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:25.871572 25100 consensus_queue.cc:260] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:25.875015 25228 tablet_copy_client.cc:308] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:25.891731 21683 consensus_queue.cc:237] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:25.899281 25209 tablet_copy_client.cc:308] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:25.905516 25100 ts_tablet_manager.cc:1398] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.066s user 0.018s sys 0.016s
I20240502 08:20:25.908056 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 12c17796e5d04014aafdac46984f4ba9 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:25.908272 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 12c17796e5d04014aafdac46984f4ba9 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-12c17796e5d04014aafdac46984f4ba9
I20240502 08:20:25.908965 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53343)
I20240502 08:20:25.909261 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53343: session id = 20b637fbec5444dd91b3af9cdda482f5-76bb1e3e2b9d4e31bf93ac1ce08a7833
I20240502 08:20:25.911451 21703 tablet_copy_source_session.cc:205] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:25.913542 21964 tablet_copy_source_session.cc:205] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:25.914570 25209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:25.914908 21550 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.915664 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-7b3f6a0b2561419ca6bd832fd65e3196 received from {username='slave'} at 127.8.68.199:43557
I20240502 08:20:25.915803 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-7b3f6a0b2561419ca6bd832fd65e3196 on tablet 7b3f6a0b2561419ca6bd832fd65e3196 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:25.915959 25150 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.918072 21943 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.919466 25034 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.922134 25209 tablet_copy_client.cc:783] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:25.922525 25209 tablet_copy_client.cc:647] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
W20240502 08:20:25.923657 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1648586b889b4a1e9ac34481cec6257a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.924126 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:25.924594 25209 tablet_copy_client.cc:523] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:25.929621 25209 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:25.932762 25231 ts_tablet_manager.cc:902] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:25.942544 21944 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:25.947516 23737 raft_consensus.cc:1203] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:25.949273 21813 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:25.950219 25149 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.953501 21683 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:20:25.958398 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec56facd7f740aeb42bc7c32495dae3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:25.958493 25228 tablet_copy_client.cc:783] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:25.958868 25228 tablet_copy_client.cc:647] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:25.962421 25228 tablet_copy_client.cc:523] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:25.965816 25149 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:25.982391 25228 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:26.006598 25252 ts_tablet_manager.cc:902] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:26.008569 25149 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.014302 25209 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.014824 25209 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:26.015146 25209 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.086s user 0.057s sys 0.006s
I20240502 08:20:26.015211 21812 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.016281 25209 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:26.016535 25209 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.016872 25209 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:26.021623 25209 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 08:20:26.027863 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-12c17796e5d04014aafdac46984f4ba9 received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:26.028030 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-12c17796e5d04014aafdac46984f4ba9 on tablet 12c17796e5d04014aafdac46984f4ba9 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:26.029386 21406 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.042928 25231 tablet_copy_client.cc:308] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:26.044747 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet efec2cc693bd4dd992d0bec22fca5419 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:20:26.044972 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet efec2cc693bd4dd992d0bec22fca5419 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-efec2cc693bd4dd992d0bec22fca5419
I20240502 08:20:26.044950 21682 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.049486 21703 tablet_copy_source_session.cc:205] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.051287 25231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efec2cc693bd4dd992d0bec22fca5419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.060652 25150 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.068439 21943 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.076644 25231 tablet_copy_client.cc:783] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.077241 25231 tablet_copy_client.cc:647] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.080622 25231 tablet_copy_client.cc:523] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.083423 21406 catalog_manager.cc:5484] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.090353 25231 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:26.105013 21552 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:26.142537 25252 tablet_copy_client.cc:308] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:26.154623 25271 ts_tablet_manager.cc:902] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:26.171819 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 182a6f2df91242cfa10f05ef788a19cb from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:20:26.172101 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 182a6f2df91242cfa10f05ef788a19cb from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-182a6f2df91242cfa10f05ef788a19cb
I20240502 08:20:26.175806 21703 tablet_copy_source_session.cc:205] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.177915 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.188805 25271 tablet_copy_client.cc:308] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:26.208772 25252 tablet_copy_client.cc:783] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.210409 25252 tablet_copy_client.cc:647] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.214241 25252 tablet_copy_client.cc:523] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.224557 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:55505)
I20240502 08:20:26.224932 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:55505: session id = 4b9bfd2541144dad86507451343640d3-9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:20:26.233397 21572 tablet_copy_source_session.cc:205] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.236107 25271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.247366 25252 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:26.247609 25271 tablet_copy_client.cc:783] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.252672 25271 tablet_copy_client.cc:647] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.256934 25228 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.257455 25228 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:26.257808 25228 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.276s user 0.051s sys 0.008s
I20240502 08:20:26.259713 25271 tablet_copy_client.cc:523] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.266183 25271 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:26.259016 25228 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:26.274662 25228 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.283432 25228 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:26.289093 25228 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20240502 08:20:26.292109 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-76bb1e3e2b9d4e31bf93ac1ce08a7833 received from {username='slave'} at 127.8.68.197:53343
I20240502 08:20:26.292327 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-76bb1e3e2b9d4e31bf93ac1ce08a7833 on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:26.326901 25231 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.327628 25231 tablet_bootstrap.cc:492] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:26.328066 25231 ts_tablet_manager.cc:1367] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.238s user 0.047s sys 0.011s
I20240502 08:20:26.329047 25231 raft_consensus.cc:348] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:26.329424 25231 raft_consensus.cc:729] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.329579 21813 consensus_queue.cc:237] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:26.329792 25231 consensus_queue.cc:260] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:26.333878 25231 ts_tablet_manager.cc:1398] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:20:26.338281 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-efec2cc693bd4dd992d0bec22fca5419 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:20:26.338482 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-efec2cc693bd4dd992d0bec22fca5419 on tablet efec2cc693bd4dd992d0bec22fca5419 with peer 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:26.340584 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 296ccf9be99545789bbb3ff4031d3170. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:26.342139 21552 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.342850 25327 ts_tablet_manager.cc:902] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:26.343458 25012 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.344055 21683 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:26.347263 21943 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.349422 25017 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.354254 23737 raft_consensus.cc:1203] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.404646 23471 raft_consensus.cc:1203] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.408537 25335 ts_tablet_manager.cc:902] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:26.410454 25012 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:26.411149 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 599d5781f1e24ae9ad425a34d88c6b68. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:26.412675 21552 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.413669 25268 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.411829 21813 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.417093 25150 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.445201 21550 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:26.446501 21407 catalog_manager.cc:5484] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:26.455793 25327 tablet_copy_client.cc:308] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:26.480960 21943 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:26.490116 25150 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:26.499598 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet d62f6eec9ee04f93a3738a83f790292c from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:57933)
I20240502 08:20:26.499830 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet d62f6eec9ee04f93a3738a83f790292c from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:57933: session id = 95f32202b9db457fb09872f6f51ea8ba-d62f6eec9ee04f93a3738a83f790292c
I20240502 08:20:26.504393 21572 tablet_copy_source_session.cc:205] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.505973 21550 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:26.507998 25327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62f6eec9ee04f93a3738a83f790292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.519210 21403 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:26.523865 25327 tablet_copy_client.cc:783] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.524962 25335 tablet_copy_client.cc:308] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:26.525061 25327 tablet_copy_client.cc:647] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.496502 25252 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.526958 25252 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:26.527330 25252 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.280s user 0.067s sys 0.003s
I20240502 08:20:26.528604 25252 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:26.528878 25252 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.531798 25271 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.532774 25271 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:26.533089 25271 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.267s user 0.066s sys 0.000s
I20240502 08:20:26.534538 25327 tablet_copy_client.cc:523] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.537638 25252 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:26.540056 25354 ts_tablet_manager.cc:902] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:26.540753 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 1648586b889b4a1e9ac34481cec6257a from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:42037)
I20240502 08:20:26.540975 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 1648586b889b4a1e9ac34481cec6257a from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:42037: session id = 95f32202b9db457fb09872f6f51ea8ba-1648586b889b4a1e9ac34481cec6257a
I20240502 08:20:26.542053 25271 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:26.546521 21703 tablet_copy_source_session.cc:205] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.547346 25327 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:26.548493 25335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1648586b889b4a1e9ac34481cec6257a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.542357 25271 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.551468 25271 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:26.543169 25252 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20240502 08:20:26.552132 25271 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:20:26.557003 25335 tablet_copy_client.cc:783] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.557509 25335 tablet_copy_client.cc:647] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.559326 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-9c0a886b3ba04737a6988c3b58c9698e received from {username='slave'} at 127.8.68.198:55505
I20240502 08:20:26.559599 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 4b9bfd2541144dad86507451343640d3-9c0a886b3ba04737a6988c3b58c9698e on tablet 9c0a886b3ba04737a6988c3b58c9698e with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:26.561038 25335 tablet_copy_client.cc:523] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.565649 25335 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:26.493466 21813 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:26.640058 25327 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.640712 25327 tablet_bootstrap.cc:492] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:26.641088 25327 ts_tablet_manager.cc:1367] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.094s user 0.062s sys 0.007s
I20240502 08:20:26.642463 25327 raft_consensus.cc:348] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:26.642844 25327 raft_consensus.cc:729] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.643182 25327 consensus_queue.cc:260] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:26.662096 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-182a6f2df91242cfa10f05ef788a19cb received from {username='slave'} at 127.8.68.198:60411
I20240502 08:20:26.662273 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-182a6f2df91242cfa10f05ef788a19cb on tablet 182a6f2df91242cfa10f05ef788a19cb with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:26.663684 23471 raft_consensus.cc:1203] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.664319 25335 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:26.664873 25335 tablet_bootstrap.cc:492] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:26.665180 25335 ts_tablet_manager.cc:1367] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.100s user 0.051s sys 0.005s
I20240502 08:20:26.666098 25335 raft_consensus.cc:348] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:26.666393 25335 raft_consensus.cc:729] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:26.666975 25335 consensus_queue.cc:260] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:26.670656 25327 ts_tablet_manager.cc:1398] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.029s user 0.019s sys 0.011s
I20240502 08:20:26.671598 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-d62f6eec9ee04f93a3738a83f790292c received from {username='slave'} at 127.8.68.200:57933
I20240502 08:20:26.671775 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-d62f6eec9ee04f93a3738a83f790292c on tablet d62f6eec9ee04f93a3738a83f790292c with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:26.683874 21683 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:26.688042 25354 tablet_copy_client.cc:308] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:26.693022 21550 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:26.693230 23603 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:26.693689 25268 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:26.693895 21944 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:20:26.695132 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182a6f2df91242cfa10f05ef788a19cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:26.695338 25269 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:26.695816 25269 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:26.704788 23602 raft_consensus.cc:1203] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.724581 21943 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:26.730901 25335 ts_tablet_manager.cc:1398] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.066s user 0.004s sys 0.007s
I20240502 08:20:26.734736 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-1648586b889b4a1e9ac34481cec6257a received from {username='slave'} at 127.8.68.200:42037
I20240502 08:20:26.734921 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-1648586b889b4a1e9ac34481cec6257a on tablet 1648586b889b4a1e9ac34481cec6257a with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:26.741410 21683 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.742091 23871 raft_consensus.cc:1203] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.742162 25149 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.744772 21813 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:26.745784 25149 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:26.756023 25268 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.762872 21550 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.767038 21403 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
W20240502 08:20:26.783810 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f5c2dfcd974ee2b8720e9eb052c3f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:26.791535 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet cec56facd7f740aeb42bc7c32495dae3 from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:57003)
I20240502 08:20:26.791798 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet cec56facd7f740aeb42bc7c32495dae3 from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:57003: session id = 95f32202b9db457fb09872f6f51ea8ba-cec56facd7f740aeb42bc7c32495dae3
I20240502 08:20:26.795970 21964 tablet_copy_source_session.cc:205] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.798174 25354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.760545 23603 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.809373 21682 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:26.810432 25252 ts_tablet_manager.cc:902] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:26.812873 21550 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:26.813616 25268 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:26.814267 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 599d5781f1e24ae9ad425a34d88c6b68. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:20:26.814548 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 599d5781f1e24ae9ad425a34d88c6b68. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:26.816980 21813 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:26.817831 25269 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:26.824586 25268 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.825618 21812 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.828433 23871 raft_consensus.cc:1203] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:26.839329 21550 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.844848 21409 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.846377 25354 tablet_copy_client.cc:783] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.846740 25354 tablet_copy_client.cc:647] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.849709 21943 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.862320 25354 tablet_copy_client.cc:523] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.872808 25413 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.874716 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:26.884568 21813 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.893992 21409 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.907600 25252 tablet_copy_client.cc:308] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:26.908515 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 296ccf9be99545789bbb3ff4031d3170 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:47771)
I20240502 08:20:26.908740 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 296ccf9be99545789bbb3ff4031d3170 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:47771: session id = 4b9bfd2541144dad86507451343640d3-296ccf9be99545789bbb3ff4031d3170
I20240502 08:20:26.908874 25354 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:26.912128 21833 tablet_copy_source_session.cc:205] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:26.914443 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296ccf9be99545789bbb3ff4031d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:26.918920 21682 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:26.932793 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:26.958010 25252 tablet_copy_client.cc:783] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:26.958382 25252 tablet_copy_client.cc:647] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:26.961500 25252 tablet_copy_client.cc:523] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:26.966441 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:27.011610 25252 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:27.066975 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:27.081128 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:27.156392 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:27.162657 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:27.168391 25354 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.169113 25354 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:27.170107 25354 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.261s user 0.056s sys 0.009s
I20240502 08:20:27.171654 25354 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:27.173156 25354 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.174192 25354 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:27.178449 25354 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:20:27.179419 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-cec56facd7f740aeb42bc7c32495dae3 received from {username='slave'} at 127.8.68.200:57003
I20240502 08:20:27.179596 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-cec56facd7f740aeb42bc7c32495dae3 on tablet cec56facd7f740aeb42bc7c32495dae3 with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:27.181255 25463 ts_tablet_manager.cc:902] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:27.194656 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:27.267393 25463 tablet_copy_client.cc:308] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:27.268476 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 182a6f2df91242cfa10f05ef788a19cb from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:27.268716 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 182a6f2df91242cfa10f05ef788a19cb from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-182a6f2df91242cfa10f05ef788a19cb
I20240502 08:20:27.272423 21703 tablet_copy_source_session.cc:205] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.277046 21682 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:27.281505 25463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:20:27.308001 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d96a7174eb5442f2b0cc0cafc3cc4c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:27.310181 21811 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.310181 21550 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.311705 25486 ts_tablet_manager.cc:902] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:27.311719 25034 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:27.313567 25150 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:27.315845 25463 tablet_copy_client.cc:783] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:27.316164 25463 tablet_copy_client.cc:647] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:27.338439 25463 tablet_copy_client.cc:523] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:27.341194 25486 tablet_copy_client.cc:308] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:27.342214 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 599d5781f1e24ae9ad425a34d88c6b68 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:20:27.342435 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 599d5781f1e24ae9ad425a34d88c6b68 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-599d5781f1e24ae9ad425a34d88c6b68
I20240502 08:20:27.346182 21703 tablet_copy_source_session.cc:205] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.348544 25486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599d5781f1e24ae9ad425a34d88c6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:27.352710 25268 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.359027 25463 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:27.362097 25486 tablet_copy_client.cc:783] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:27.362955 21550 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.372792 21409 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.376019 25486 tablet_copy_client.cc:647] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:27.383210 25486 tablet_copy_client.cc:523] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:27.387135 25486 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:27.385531 21811 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.388871 25497 ts_tablet_manager.cc:902] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:27.438149 25497 tablet_copy_client.cc:308] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:27.451300 21943 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:27.455108 21811 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:27.455163 21682 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:27.455818 25422 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:27.456097 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f5c2dfcd974ee2b8720e9eb052c3f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:27.456301 25520 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:27.458765 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f5c2dfcd974ee2b8720e9eb052c3f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:27.459892 25513 ts_tablet_manager.cc:902] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:27.459535 25422 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.466746 21683 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.475554 21403 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.476886 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:20:27.477058 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:20:27.478992 25252 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.479496 25252 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:27.479787 25252 ts_tablet_manager.cc:1367] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.468s user 0.053s sys 0.010s
I20240502 08:20:27.487665 25252 raft_consensus.cc:348] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:27.489385 25252 raft_consensus.cc:729] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.490082 25252 consensus_queue.cc:260] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:27.497331 21964 tablet_copy_source_session.cc:205] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.499356 25497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:27.503348 25252 ts_tablet_manager.cc:1398] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240502 08:20:27.507423 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-296ccf9be99545789bbb3ff4031d3170 received from {username='slave'} at 127.8.68.198:47771
I20240502 08:20:27.507573 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 4b9bfd2541144dad86507451343640d3-296ccf9be99545789bbb3ff4031d3170 on tablet 296ccf9be99545789bbb3ff4031d3170 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:27.514569 25497 tablet_copy_client.cc:783] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:27.514911 21811 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.515203 25497 tablet_copy_client.cc:647] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:27.520977 25497 tablet_copy_client.cc:523] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:27.525391 25497 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:27.573593 25463 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.573814 25513 tablet_copy_client.cc:308] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:27.576488 25463 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:27.576831 25463 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.218s user 0.059s sys 0.000s
I20240502 08:20:27.577082 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 599d5781f1e24ae9ad425a34d88c6b68 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:27.577271 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 599d5781f1e24ae9ad425a34d88c6b68 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-599d5781f1e24ae9ad425a34d88c6b68
I20240502 08:20:27.577673 25463 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.577944 25463 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.578233 25463 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.580775 21703 tablet_copy_source_session.cc:205] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.591969 25486 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.592708 25486 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:27.593216 25486 ts_tablet_manager.cc:1367] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.206s user 0.062s sys 0.007s
I20240502 08:20:27.594529 25486 raft_consensus.cc:348] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.595010 25486 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.595530 25486 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.603372 25486 ts_tablet_manager.cc:1398] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:20:27.609016 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599d5781f1e24ae9ad425a34d88c6b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:27.610677 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-599d5781f1e24ae9ad425a34d88c6b68 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:20:27.610859 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-599d5781f1e24ae9ad425a34d88c6b68 on tablet 599d5781f1e24ae9ad425a34d88c6b68 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:27.614471 25463 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.038s user 0.003s sys 0.015s
I20240502 08:20:27.615697 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-182a6f2df91242cfa10f05ef788a19cb received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:27.615835 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-182a6f2df91242cfa10f05ef788a19cb on tablet 182a6f2df91242cfa10f05ef788a19cb with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:27.627233 25513 tablet_copy_client.cc:783] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:27.641036 25513 tablet_copy_client.cc:647] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:27.647923 25513 tablet_copy_client.cc:523] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:27.655747 25513 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:27.667233 23871 raft_consensus.cc:1203] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:27.687961 25252 ts_tablet_manager.cc:902] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:27.702313 25497 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.702883 25497 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:27.703183 25497 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.178s user 0.057s sys 0.031s
I20240502 08:20:27.712625 25497 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:27.713052 25497 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.714093 25497 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:27.715384 25497 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:20:27.716235 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:20:27.716395 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7 on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:27.721233 23471 raft_consensus.cc:1203] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:27.723320 25252 tablet_copy_client.cc:308] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:27.724179 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:20:27.724350 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-d96a7174eb5442f2b0cc0cafc3cc4c80
I20240502 08:20:27.731235 21703 tablet_copy_source_session.cc:205] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.735189 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:27.746623 25252 tablet_copy_client.cc:783] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:27.750942 25252 tablet_copy_client.cc:647] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:27.753808 21683 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:27.755489 21550 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:27.763626 25252 tablet_copy_client.cc:523] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:20:27.768131 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0241228bd204ca2a8ad2c57b6eeb543. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:27.769416 21682 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.770331 25326 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:27.772967 25513 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:27.773600 25513 tablet_bootstrap.cc:492] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:27.774024 25513 ts_tablet_manager.cc:1367] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.118s user 0.054s sys 0.016s
I20240502 08:20:27.775650 25513 raft_consensus.cc:348] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.775974 25513 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:27.776289 25513 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:27.777109 25513 ts_tablet_manager.cc:1398] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:20:27.779476 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-599d5781f1e24ae9ad425a34d88c6b68 received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:27.779670 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-599d5781f1e24ae9ad425a34d88c6b68 on tablet 599d5781f1e24ae9ad425a34d88c6b68 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:27.785801 21811 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.787012 25326 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:27.799381 21618 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:27.800789 21550 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.801561 25034 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:27.818661 21943 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:27.820106 23737 raft_consensus.cc:1203] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:27.820236 25268 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:27.836854 25470 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.842440 21811 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.845525 21682 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.849743 21409 catalog_manager.cc:5484] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:27.850170 25268 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.858351 21552 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.868350 25513 tablet_replica.cc:330] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: stopping tablet replica
I20240502 08:20:27.869158 25513 raft_consensus.cc:2229] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:27.869637 25513 raft_consensus.cc:2258] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:27.871697 25513 ts_tablet_manager.cc:1875] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:27.876684 21403 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.876927 21944 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:27.836086 25252 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:27.906555 25513 ts_tablet_manager.cc:1888] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:27.906826 25513 log.cc:1196] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/wals/07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:20:27.914456 25513 ts_tablet_manager.cc:902] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:27.922230 23603 raft_consensus.cc:1203] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:27.935604 25486 ts_tablet_manager.cc:902] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:27.954138 25513 tablet_copy_client.cc:308] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:27.955204 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:20:27.955500 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:20:27.963644 21964 tablet_copy_source_session.cc:205] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.966080 25513 ts_tablet_manager.cc:1875] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:20:27.988111 25486 tablet_copy_client.cc:308] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:27.989137 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53343)
I20240502 08:20:27.989362 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53343: session id = 20b637fbec5444dd91b3af9cdda482f5-07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:20:27.993811 21964 tablet_copy_source_session.cc:205] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:27.998314 25486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:28.016983 25513 ts_tablet_manager.cc:1888] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:20:28.017406 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:28.022646 25486 tablet_copy_client.cc:783] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:28.022951 25486 tablet_copy_client.cc:647] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:28.024797 25486 tablet_copy_client.cc:523] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:28.046531 25513 tablet_copy_client.cc:783] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:28.046846 25513 tablet_copy_client.cc:647] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:28.048586 25486 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:28.055315 25513 tablet_copy_client.cc:523] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:28.059532 25513 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:28.121316 21811 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:20:28.136397 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de41d6c6de0400397645c05a0fa17de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:28.137976 21552 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.138139 21944 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.138929 25012 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.139695 25012 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.157644 25252 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.158236 25252 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:28.158628 25252 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.323s user 0.061s sys 0.005s
I20240502 08:20:28.168442 25017 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:28.176383 25252 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:28.176800 25252 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.177179 25252 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:28.176537 21552 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:28.180220 25252 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20240502 08:20:28.181216 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-d96a7174eb5442f2b0cc0cafc3cc4c80 received from {username='slave'} at 127.8.68.198:60411
I20240502 08:20:28.181406 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-d96a7174eb5442f2b0cc0cafc3cc4c80 on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:28.183341 23737 raft_consensus.cc:1203] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:28.194132 21407 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:28.218936 21944 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:28.227156 25486 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.227969 25486 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:28.228355 25486 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.185s user 0.059s sys 0.006s
I20240502 08:20:28.229668 25486 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.230021 25486 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.230406 25486 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.249262 25661 ts_tablet_manager.cc:902] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:28.255851 25661 tablet_copy_client.cc:308] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:28.257462 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:20:28.257741 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:20:28.262235 21703 tablet_copy_source_session.cc:205] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:28.282419 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:28.302264 25486 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.074s user 0.013s sys 0.025s
I20240502 08:20:28.303792 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-07f5c2dfcd974ee2b8720e9eb052c3f7 received from {username='slave'} at 127.8.68.197:53343
I20240502 08:20:28.304018 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-07f5c2dfcd974ee2b8720e9eb052c3f7 on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:28.345713 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:28.360736 25252 ts_tablet_manager.cc:902] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:28.425839 23471 raft_consensus.cc:1203] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:28.435912 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:28.445463 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:28.486200 25661 tablet_copy_client.cc:783] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:28.486732 25661 tablet_copy_client.cc:647] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:28.489665 25661 tablet_copy_client.cc:523] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:28.496803 25661 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
W20240502 08:20:28.531023 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 52335.9us
I20240502 08:20:28.531426 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:28.551435 25513 log.cc:924] Time spent T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240502 08:20:28.554419 25718 ts_tablet_manager.cc:902] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:28.557292 25252 tablet_copy_client.cc:308] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:28.558146 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet e0241228bd204ca2a8ad2c57b6eeb543 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:55505)
I20240502 08:20:28.558377 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet e0241228bd204ca2a8ad2c57b6eeb543 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:55505: session id = 4b9bfd2541144dad86507451343640d3-e0241228bd204ca2a8ad2c57b6eeb543
I20240502 08:20:28.561630 21572 tablet_copy_source_session.cc:205] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:28.579213 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:28.579532 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:28.581526 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:28.583592 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0241228bd204ca2a8ad2c57b6eeb543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:28.592617 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:28.602911 21682 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:28.612823 25513 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.619045 23603 raft_consensus.cc:1203] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:28.636068 25513 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
W20240502 08:20:28.641728 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20b2fd9f1ce488ba7793ed7ac268c16. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:28.643627 21944 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.644428 21550 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.653038 25595 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.653867 25595 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.656246 25252 tablet_copy_client.cc:783] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:28.656545 25252 tablet_copy_client.cc:647] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:28.659967 21811 consensus_queue.cc:237] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.671308 25513 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.577s user 0.065s sys 0.000s
I20240502 08:20:28.672362 25513 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.672833 25513 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.673266 25513 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.683977 25513 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:20:28.685544 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:20:28.685743 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-07f5c2dfcd974ee2b8720e9eb052c3f7 on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:28.690909 25718 tablet_copy_client.cc:308] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:28.691761 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 2de41d6c6de0400397645c05a0fa17de from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:43557)
I20240502 08:20:28.691953 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 2de41d6c6de0400397645c05a0fa17de from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:43557: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-2de41d6c6de0400397645c05a0fa17de
I20240502 08:20:28.695907 21833 tablet_copy_source_session.cc:205] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:28.699995 25718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:28.724691 21682 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.725735 25012 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.726033 21943 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:28.726792 25012 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:28.772094 25252 tablet_copy_client.cc:523] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:28.784070 25252 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:28.786566 25012 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:28.788033 25729 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:28.792136 23737 raft_consensus.cc:1203] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:28.793332 21552 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:28.797431 21944 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:28.801879 21409 catalog_manager.cc:5484] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:28.808429 21682 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:28.824725 25718 tablet_copy_client.cc:783] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:28.825091 25718 tablet_copy_client.cc:647] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:28.820617 21944 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:28.829404 25718 tablet_copy_client.cc:523] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:28.840571 25718 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:28.844827 21406 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:28.919912 25661 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.921685 25661 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:28.922106 25661 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.425s user 0.062s sys 0.002s
I20240502 08:20:28.923931 25661 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.930043 25661 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.930418 25661 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:28.931195 25661 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:20:28.932631 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-6f04a3972e384fa3b0ffe0cb8a4f35aa received from {username='slave'} at 127.8.68.197:35657
I20240502 08:20:28.932894 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-6f04a3972e384fa3b0ffe0cb8a4f35aa on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with peer 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:28.951189 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8042f6ab6404c5a9f8cce0b8f971542. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:28.980240 25718 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.980826 25718 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:28.981184 25718 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.141s user 0.046s sys 0.011s
I20240502 08:20:28.982167 25718 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:28.982451 25718 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.982779 25718 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:28.987430 25718 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:20:28.988744 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-2de41d6c6de0400397645c05a0fa17de received from {username='slave'} at 127.8.68.199:43557
I20240502 08:20:28.988936 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-2de41d6c6de0400397645c05a0fa17de on tablet 2de41d6c6de0400397645c05a0fa17de with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:28.989671 25252 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:28.990190 25252 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:28.990501 25252 ts_tablet_manager.cc:1367] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.207s user 0.084s sys 0.042s
I20240502 08:20:28.992333 25252 raft_consensus.cc:348] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:28.992605 25252 raft_consensus.cc:729] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:28.992965 25252 consensus_queue.cc:260] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:28.993633 25252 ts_tablet_manager.cc:1398] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 08:20:28.994719 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-e0241228bd204ca2a8ad2c57b6eeb543 received from {username='slave'} at 127.8.68.198:55505
I20240502 08:20:28.994912 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 4b9bfd2541144dad86507451343640d3-e0241228bd204ca2a8ad2c57b6eeb543 on tablet e0241228bd204ca2a8ad2c57b6eeb543 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:29.036226 23471 raft_consensus.cc:1203] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:29.060864 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:29.073958 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:29.131867 25777 ts_tablet_manager.cc:902] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:29.215921 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:29.224891 25661 ts_tablet_manager.cc:902] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:29.242877 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:29.272701 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.307149 21811 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:20:29.319640 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc49c050280346db88d1323e30513c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:29.321841 21552 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:29.322849 25017 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:29.329298 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.338963 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.364408 23737 raft_consensus.cc:1203] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:29.412519 23603 raft_consensus.cc:1203] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:29.452339 21944 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:29.463016 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:29.470737 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:29.506635 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.519284 25777 tablet_copy_client.cc:308] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:29.520474 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:42037)
I20240502 08:20:29.520782 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:42037: session id = 95f32202b9db457fb09872f6f51ea8ba-a20b2fd9f1ce488ba7793ed7ac268c16
I20240502 08:20:29.526971 21703 tablet_copy_source_session.cc:205] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:29.586747 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:29.587857 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:29.589617 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 08:20:29.610289 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 219494us
I20240502 08:20:29.614281 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.615128 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.617695 25012 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:29.638345 25777 tablet_copy_client.cc:783] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:29.642522 25777 tablet_copy_client.cc:647] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:29.653609 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.659471 25777 tablet_copy_client.cc:523] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:29.663977 25777 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
W20240502 08:20:29.674412 21487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 52896.8us
I20240502 08:20:29.746089 25817 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:29.751641 25661 tablet_copy_client.cc:308] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:29.752228 21943 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:29.753986 21550 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:29.762022 21411 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:29.700026 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:29.848839 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet d8042f6ab6404c5a9f8cce0b8f971542 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53979)
I20240502 08:20:29.849148 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet d8042f6ab6404c5a9f8cce0b8f971542 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53979: session id = 20b637fbec5444dd91b3af9cdda482f5-d8042f6ab6404c5a9f8cce0b8f971542
I20240502 08:20:29.854651 21833 tablet_copy_source_session.cc:205] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:29.856580 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8042f6ab6404c5a9f8cce0b8f971542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:29.869148 25661 tablet_copy_client.cc:783] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:29.869467 25661 tablet_copy_client.cc:647] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:29.872553 25661 tablet_copy_client.cc:523] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:29.873376 25835 ts_tablet_manager.cc:902] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:29.877497 25661 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:29.952052 25835 tablet_copy_client.cc:308] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:29.953056 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet dc49c050280346db88d1323e30513c80 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:43557)
I20240502 08:20:29.953276 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet dc49c050280346db88d1323e30513c80 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:43557: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-dc49c050280346db88d1323e30513c80
I20240502 08:20:29.956670 21833 tablet_copy_source_session.cc:205] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:29.958272 25835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:29.967190 25835 tablet_copy_client.cc:783] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:29.967885 25835 tablet_copy_client.cc:647] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:29.969977 25835 tablet_copy_client.cc:523] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:29.974864 25835 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:30.002384 25661 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:30.003188 25661 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:30.003536 25661 ts_tablet_manager.cc:1367] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.126s user 0.047s sys 0.011s
I20240502 08:20:30.004370 25661 raft_consensus.cc:348] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:30.004702 25661 raft_consensus.cc:729] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:30.005021 25661 consensus_queue.cc:260] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:30.008774 21682 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
W20240502 08:20:30.020222 21616 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.025100 23471 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.026152 25595 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.028529 21943 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.029289 25595 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.037420 21550 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.038344 25595 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.082751 25595 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.092656 21405 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.090868 23471 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.093904 25835 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:30.094774 25835 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:30.095175 25835 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.120s user 0.045s sys 0.007s
I20240502 08:20:30.096176 25835 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:30.096475 25835 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:30.096747 25835 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:30.090700 21550 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.123082 25661 ts_tablet_manager.cc:1398] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.119s user 0.019s sys 0.019s
I20240502 08:20:30.124217 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d8042f6ab6404c5a9f8cce0b8f971542 received from {username='slave'} at 127.8.68.197:53979
I20240502 08:20:30.124387 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d8042f6ab6404c5a9f8cce0b8f971542 on tablet d8042f6ab6404c5a9f8cce0b8f971542 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:30.124879 21944 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.153272 25835 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.058s user 0.008s sys 0.009s
I20240502 08:20:30.154143 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-dc49c050280346db88d1323e30513c80 received from {username='slave'} at 127.8.68.199:43557
I20240502 08:20:30.154314 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-dc49c050280346db88d1323e30513c80 on tablet dc49c050280346db88d1323e30513c80 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:30.228744 23471 raft_consensus.cc:1203] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:30.274159 21682 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:20:30.285427 21618 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12c17796e5d04014aafdac46984f4ba9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.286975 23737 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.287895 25598 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.289920 21550 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.290633 25595 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.295065 21811 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.296324 25595 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.300181 25777 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:30.301728 25777 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:30.302078 25777 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.638s user 0.051s sys 0.008s
I20240502 08:20:30.302906 25777 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:30.303174 25777 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:30.305768 25777 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:30.308948 23736 raft_consensus.cc:1203] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:30.327105 25777 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.025s user 0.003s sys 0.002s
I20240502 08:20:30.328367 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-a20b2fd9f1ce488ba7793ed7ac268c16 received from {username='slave'} at 127.8.68.200:42037
I20240502 08:20:30.328562 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-a20b2fd9f1ce488ba7793ed7ac268c16 on tablet a20b2fd9f1ce488ba7793ed7ac268c16 with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:30.343235 25595 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.349270 21813 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.352283 23737 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.360852 21411 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.362744 21550 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.429049 21682 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:30.433125 21811 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:30.434541 25900 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:30.435673 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b88c4a1a36424086aad89b7585c17960. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.436522 21943 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:30.438858 25900 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:30.455412 25598 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.460557 21943 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.463650 21811 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.466691 21405 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.497514 25777 ts_tablet_manager.cc:902] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:30.543911 25777 tablet_copy_client.cc:308] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:30.544811 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:42037)
I20240502 08:20:30.545014 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:42037: session id = 95f32202b9db457fb09872f6f51ea8ba-6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:20:30.548772 21703 tablet_copy_source_session.cc:205] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:30.551165 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:30.568044 25777 tablet_copy_client.cc:783] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:30.568378 25777 tablet_copy_client.cc:647] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:30.570997 25777 tablet_copy_client.cc:523] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:30.573907 23871 raft_consensus.cc:1203] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:30.579638 25777 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:30.673559 21811 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:30.681757 25777 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:30.682439 25777 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:30.682806 25777 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.103s user 0.076s sys 0.009s
I20240502 08:20:30.684036 25777 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:30.684419 25777 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:30.684860 25777 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:30.687278 21813 consensus_queue.cc:237] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:30.689240 25777 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:20:30.695808 21943 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.696796 25012 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.699822 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-6f04a3972e384fa3b0ffe0cb8a4f35aa received from {username='slave'} at 127.8.68.200:42037
I20240502 08:20:30.699965 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-6f04a3972e384fa3b0ffe0cb8a4f35aa on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:30.701347 21682 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.703486 23471 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:30.704351 25817 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:30.705081 25817 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:30.713970 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8042f6ab6404c5a9f8cce0b8f971542. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.717074 25012 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.720665 23471 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.726569 21813 consensus_queue.cc:237] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:30.730288 21943 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:30.738025 21748 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 203e7914da3f4494b239c6603513caee. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.739789 21550 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:30.741951 21944 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:30.743463 25817 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:30.745071 25993 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:30.745687 21406 catalog_manager.cc:5484] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.754204 21552 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:20:30.775061 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4862ea2f67c43a28689b464d8ec96e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:30.775430 25993 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:30.776333 26007 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.780833 21551 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.781373 21943 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:30.782215 25817 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:30.787143 25817 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.791644 21406 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.793435 21943 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.800877 21406 catalog_manager.cc:5484] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.803773 21551 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:30.817718 21682 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:30.850243 21943 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:30.874373 26019 ts_tablet_manager.cc:902] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:30.896567 26042 ts_tablet_manager.cc:902] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:30.919036 26019 tablet_copy_client.cc:308] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:30.919957 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 12c17796e5d04014aafdac46984f4ba9 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:20:30.920310 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 12c17796e5d04014aafdac46984f4ba9 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-12c17796e5d04014aafdac46984f4ba9
I20240502 08:20:30.934696 26042 tablet_copy_client.cc:308] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:30.936379 21702 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet b88c4a1a36424086aad89b7585c17960 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:20:30.936653 21702 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet b88c4a1a36424086aad89b7585c17960 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-b88c4a1a36424086aad89b7585c17960
I20240502 08:20:30.941015 21703 tablet_copy_source_session.cc:205] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:30.941696 21702 tablet_copy_source_session.cc:205] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:30.943779 26042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:30.948060 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:30.949550 26042 tablet_copy_client.cc:783] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:30.949858 26042 tablet_copy_client.cc:647] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:30.951803 26042 tablet_copy_client.cc:523] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:30.970342 26042 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:30.973553 26019 tablet_copy_client.cc:783] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:30.973989 26019 tablet_copy_client.cc:647] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:30.976536 26019 tablet_copy_client.cc:523] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:30.980777 26019 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:30.995194 21682 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:30.999752 23471 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:31.000810 25900 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:31.001266 21943 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 08:20:31.002161 21618 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:31.002755 25900 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:31.016454 23871 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:31.018112 25900 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:31.034327 25595 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.040745 23870 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.063670 21406 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.083259 21550 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:31.085038 25900 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:31.091480 21550 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.092518 21551 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:31.097805 23603 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.099902 21943 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.101308 25713 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:31.134186 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0241228bd204ca2a8ad2c57b6eeb543. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:31.136699 21682 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.137938 21811 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.139526 23471 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.144081 25713 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:31.144887 25713 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:31.180130 25713 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.194303 21811 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.196055 26096 ts_tablet_manager.cc:902] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:31.198701 23602 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.209820 26097 ts_tablet_manager.cc:902] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:31.210060 21411 catalog_manager.cc:5484] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.236325 26096 tablet_copy_client.cc:308] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:31.237447 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 203e7914da3f4494b239c6603513caee from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53979)
I20240502 08:20:31.237677 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 203e7914da3f4494b239c6603513caee from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53979: session id = 20b637fbec5444dd91b3af9cdda482f5-203e7914da3f4494b239c6603513caee
I20240502 08:20:31.240721 26097 tablet_copy_client.cc:308] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:31.240830 21833 tablet_copy_source_session.cc:205] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:31.242105 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet d8042f6ab6404c5a9f8cce0b8f971542 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:47771)
I20240502 08:20:31.242383 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet d8042f6ab6404c5a9f8cce0b8f971542 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:47771: session id = 4b9bfd2541144dad86507451343640d3-d8042f6ab6404c5a9f8cce0b8f971542
I20240502 08:20:31.242954 26096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e7914da3f4494b239c6603513caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:31.245847 26115 ts_tablet_manager.cc:902] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:31.245970 21833 tablet_copy_source_session.cc:205] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:31.247684 26097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8042f6ab6404c5a9f8cce0b8f971542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:31.251224 21682 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:31.284142 26096 tablet_copy_client.cc:783] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:31.284523 26096 tablet_copy_client.cc:647] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:31.289638 26096 tablet_copy_client.cc:523] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:31.293558 26097 tablet_copy_client.cc:783] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:31.293982 26097 tablet_copy_client.cc:647] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:31.299351 26097 tablet_copy_client.cc:523] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:31.314205 26096 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:31.314445 26097 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:31.322618 21682 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:31.340281 26115 tablet_copy_client.cc:308] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:31.341234 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet c4862ea2f67c43a28689b464d8ec96e6 from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:35393)
I20240502 08:20:31.341464 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet c4862ea2f67c43a28689b464d8ec96e6 from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:35393: session id = 95f32202b9db457fb09872f6f51ea8ba-c4862ea2f67c43a28689b464d8ec96e6
I20240502 08:20:31.345387 21833 tablet_copy_source_session.cc:205] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:31.348264 26115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4862ea2f67c43a28689b464d8ec96e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:20:31.347900 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 44141.8us
I20240502 08:20:31.355659 21943 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:31.356370 21811 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:31.357220 26042 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.357323 25900 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:31.357810 26042 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:31.358191 26042 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.388s user 0.069s sys 0.031s
I20240502 08:20:31.359902 26075 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:31.360577 26042 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:31.360918 26042 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
W20240502 08:20:31.362346 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a45b1da348e4bb89f021df103a0603f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:31.361222 26042 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:31.369457 26042 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:20:31.374511 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-b88c4a1a36424086aad89b7585c17960 received from {username='slave'} at 127.8.68.198:60411
I20240502 08:20:31.374752 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-b88c4a1a36424086aad89b7585c17960 on tablet b88c4a1a36424086aad89b7585c17960 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:31.380271 26115 tablet_copy_client.cc:783] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:31.380673 26115 tablet_copy_client.cc:647] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:31.384883 26115 tablet_copy_client.cc:523] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:31.401942 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:31.410053 26115 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:31.417392 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:31.427234 26075 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:31.432134 21813 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:31.435642 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:31.437831 23602 raft_consensus.cc:1203] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:31.442081 21405 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:31.501255 26157 ts_tablet_manager.cc:902] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:31.501603 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:31.502791 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:31.519515 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:31.524384 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:31.503669 21943 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:31.577282 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:31.589715 26178 ts_tablet_manager.cc:902] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:31.612504 26097 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.613065 26097 tablet_bootstrap.cc:492] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:31.613384 26097 ts_tablet_manager.cc:1367] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.299s user 0.070s sys 0.035s
I20240502 08:20:31.614364 26097 raft_consensus.cc:348] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:31.614729 26097 raft_consensus.cc:729] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:31.615026 26097 consensus_queue.cc:260] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:31.615772 26097 ts_tablet_manager.cc:1398] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:31.616712 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-d8042f6ab6404c5a9f8cce0b8f971542 received from {username='slave'} at 127.8.68.198:47771
I20240502 08:20:31.616885 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 4b9bfd2541144dad86507451343640d3-d8042f6ab6404c5a9f8cce0b8f971542 on tablet d8042f6ab6404c5a9f8cce0b8f971542 with peer 4b9bfd2541144dad86507451343640d3
W20240502 08:20:31.620888 26019 log.cc:924] Time spent T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Append to log took a long time: real 0.058s user 0.004s sys 0.000s
I20240502 08:20:31.623312 26019 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.628489 26019 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:31.642850 26019 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.662s user 0.063s sys 0.017s
I20240502 08:20:31.646035 23602 raft_consensus.cc:1203] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:31.646013 26019 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:31.646309 26019 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:31.646734 26157 tablet_copy_client.cc:308] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:31.647737 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:31.647954 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:20:31.651185 21703 tablet_copy_source_session.cc:205] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:31.656469 26157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:31.665417 26019 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:31.670857 21811 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
W20240502 08:20:31.681309 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de41d6c6de0400397645c05a0fa17de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:31.681775 26157 tablet_copy_client.cc:783] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:31.681416 26096 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.682081 21551 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.683606 21943 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.683831 26019 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.039s user 0.003s sys 0.001s
I20240502 08:20:31.684248 26163 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:31.684579 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-12c17796e5d04014aafdac46984f4ba9 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:20:31.684716 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-12c17796e5d04014aafdac46984f4ba9 on tablet 12c17796e5d04014aafdac46984f4ba9 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:31.684957 26163 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:31.686894 26157 tablet_copy_client.cc:647] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:31.687395 26096 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:31.688004 26096 ts_tablet_manager.cc:1367] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.374s user 0.050s sys 0.011s
I20240502 08:20:31.689394 26096 raft_consensus.cc:348] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:31.690120 26096 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:31.690550 26096 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:31.693013 26096 ts_tablet_manager.cc:1398] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:20:31.693842 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-203e7914da3f4494b239c6603513caee received from {username='slave'} at 127.8.68.197:53979
I20240502 08:20:31.693981 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-203e7914da3f4494b239c6603513caee on tablet 203e7914da3f4494b239c6603513caee with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:31.723137 23470 raft_consensus.cc:1203] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:31.724766 23737 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:31.725589 26009 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:31.728193 26009 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:31.734490 21551 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:31.741225 21406 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:31.743310 26157 tablet_copy_client.cc:523] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:31.745827 21943 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:31.747140 26157 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:31.749785 26178 tablet_copy_client.cc:308] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:31.776042 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet e0241228bd204ca2a8ad2c57b6eeb543 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:36105)
I20240502 08:20:31.776304 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet e0241228bd204ca2a8ad2c57b6eeb543 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:36105: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-e0241228bd204ca2a8ad2c57b6eeb543
I20240502 08:20:31.779857 21572 tablet_copy_source_session.cc:205] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:31.781486 26178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0241228bd204ca2a8ad2c57b6eeb543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:31.789717 26178 tablet_copy_client.cc:783] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:31.790052 26178 tablet_copy_client.cc:647] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:31.795181 26178 tablet_copy_client.cc:523] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:31.805907 26178 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:31.814244 23737 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:31.889127 23470 raft_consensus.cc:1203] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:31.899921 26157 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.900377 21551 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:31.908391 26097 ts_tablet_manager.cc:902] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:31.921401 21943 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:31.922396 26101 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:31.931335 21811 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:31.932313 26101 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:31.944849 26115 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:31.945573 26115 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:31.945991 26115 ts_tablet_manager.cc:1367] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.536s user 0.065s sys 0.000s
I20240502 08:20:31.953670 26101 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:31.956773 21944 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:31.969806 21405 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:31.975270 21811 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:20:31.986594 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f5d97e6e0a4755be41699246907e55. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:31.986821 26115 raft_consensus.cc:348] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:31.987138 26115 raft_consensus.cc:729] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:31.987619 26115 consensus_queue.cc:260] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:32.044509 26157 tablet_bootstrap.cc:492] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:32.044974 26157 ts_tablet_manager.cc:1367] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.298s user 0.049s sys 0.010s
I20240502 08:20:32.071879 26115 ts_tablet_manager.cc:1398] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.126s user 0.018s sys 0.006s
I20240502 08:20:32.072655 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-c4862ea2f67c43a28689b464d8ec96e6 received from {username='slave'} at 127.8.68.200:35393
I20240502 08:20:32.072834 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-c4862ea2f67c43a28689b464d8ec96e6 on tablet c4862ea2f67c43a28689b464d8ec96e6 with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:32.046078 26157 raft_consensus.cc:348] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:32.081970 26178 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:32.093137 26178 tablet_bootstrap.cc:492] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:32.075613 26157 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:32.093963 26157 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:32.095259 26157 ts_tablet_manager.cc:1398] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20240502 08:20:32.096263 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-6f04a3972e384fa3b0ffe0cb8a4f35aa received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:32.096449 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-6f04a3972e384fa3b0ffe0cb8a4f35aa on tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:32.097317 26178 ts_tablet_manager.cc:1367] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.292s user 0.060s sys 0.005s
I20240502 08:20:32.098438 26178 raft_consensus.cc:348] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:32.098990 26178 raft_consensus.cc:729] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:32.099467 26178 consensus_queue.cc:260] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:32.100648 26178 ts_tablet_manager.cc:1398] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:32.105355 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-e0241228bd204ca2a8ad2c57b6eeb543 received from {username='slave'} at 127.8.68.199:36105
I20240502 08:20:32.105501 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-e0241228bd204ca2a8ad2c57b6eeb543 on tablet e0241228bd204ca2a8ad2c57b6eeb543 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:32.131776 26097 tablet_copy_client.cc:308] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:32.132858 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 8a45b1da348e4bb89f021df103a0603f from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:20:32.133085 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 8a45b1da348e4bb89f021df103a0603f from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-8a45b1da348e4bb89f021df103a0603f
I20240502 08:20:32.136039 26115 ts_tablet_manager.cc:902] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:32.137099 21703 tablet_copy_source_session.cc:205] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:32.140364 26097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45b1da348e4bb89f021df103a0603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:32.164286 26097 tablet_copy_client.cc:783] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:32.164588 26097 tablet_copy_client.cc:647] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:32.167402 26097 tablet_copy_client.cc:523] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:32.174021 23737 raft_consensus.cc:1203] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:32.182968 26097 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:32.200834 26115 tablet_copy_client.cc:308] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:32.201635 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 2de41d6c6de0400397645c05a0fa17de from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:35393)
I20240502 08:20:32.201900 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 2de41d6c6de0400397645c05a0fa17de from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:35393: session id = 95f32202b9db457fb09872f6f51ea8ba-2de41d6c6de0400397645c05a0fa17de
I20240502 08:20:32.204932 21833 tablet_copy_source_session.cc:205] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:32.208402 26115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:32.224938 21550 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:32.225870 23871 raft_consensus.cc:1203] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 08:20:32.229653 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f5d97e6e0a4755be41699246907e55. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.231802 21811 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:20:32.232651 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f5d97e6e0a4755be41699246907e55. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.232868 26101 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.235293 26115 tablet_copy_client.cc:783] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:32.235589 26115 tablet_copy_client.cc:647] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:32.240700 26115 tablet_copy_client.cc:523] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:32.246763 26115 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:32.248386 21944 consensus_queue.cc:237] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:20:32.268183 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b3ba1d834a4314b63f336675845441. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.269023 21811 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:32.270557 25934 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:32.280336 21682 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:32.281064 25934 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:32.292498 21944 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.293380 26101 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.298213 26101 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:32.303979 21943 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:32.309341 21812 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:32.318859 21411 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:32.325531 21812 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:32.352411 26135 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.354606 26097 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:32.355113 26097 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:32.355432 26097 ts_tablet_manager.cc:1367] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.173s user 0.060s sys 0.002s
I20240502 08:20:32.356344 26097 raft_consensus.cc:348] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:32.356581 26097 raft_consensus.cc:729] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:32.356878 26097 consensus_queue.cc:260] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:32.359176 21682 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.369865 23737 raft_consensus.cc:1203] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:20:32.369736 21405 catalog_manager.cc:5484] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.372143 26097 ts_tablet_manager.cc:1398] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:20:32.381182 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-8a45b1da348e4bb89f021df103a0603f received from {username='slave'} at 127.8.68.198:60411
I20240502 08:20:32.381402 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-8a45b1da348e4bb89f021df103a0603f on tablet 8a45b1da348e4bb89f021df103a0603f with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:32.382568 23602 raft_consensus.cc:1203] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:32.388064 21811 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.417035 23737 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.417188 21944 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.417855 26009 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.421296 26009 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:32.423668 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc49c050280346db88d1323e30513c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.466022 26307 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:32.467375 21550 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.468533 26009 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.471045 21550 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:32.474495 23737 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:32.478968 21944 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:32.480641 21411 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:32.485472 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:32.522861 26115 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:32.523830 26115 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:32.524153 26115 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.277s user 0.072s sys 0.034s
I20240502 08:20:32.525079 26115 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:32.525360 26115 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:32.525678 26115 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:32.526790 26115 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:32.530570 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-2de41d6c6de0400397645c05a0fa17de received from {username='slave'} at 127.8.68.200:35393
I20240502 08:20:32.530804 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-2de41d6c6de0400397645c05a0fa17de on tablet 2de41d6c6de0400397645c05a0fa17de with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:32.559664 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:32.560191 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:32.583978 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:32.592978 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:32.598104 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:32.612939 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 83271.6us
I20240502 08:20:32.616925 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:32.629231 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:32.653753 23871 raft_consensus.cc:1203] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:32.740095 26347 ts_tablet_manager.cc:902] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:32.783177 26347 tablet_copy_client.cc:308] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:32.816195 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 59f5d97e6e0a4755be41699246907e55 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:52703)
I20240502 08:20:32.816421 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 59f5d97e6e0a4755be41699246907e55 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:52703: session id = 20b637fbec5444dd91b3af9cdda482f5-59f5d97e6e0a4755be41699246907e55
I20240502 08:20:32.825354 26097 ts_tablet_manager.cc:902] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:32.826294 21572 tablet_copy_source_session.cc:205] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:32.827904 26347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:32.833578 26365 ts_tablet_manager.cc:902] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:32.859452 26347 tablet_copy_client.cc:783] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:32.860347 26347 tablet_copy_client.cc:647] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:32.866200 26347 tablet_copy_client.cc:523] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:32.870652 21944 consensus_queue.cc:237] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:20:32.876072 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b3ba1d834a4314b63f336675845441. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.876796 21683 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.876821 26365 tablet_copy_client.cc:308] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:32.877559 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 80b3ba1d834a4314b63f336675845441 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:20:32.877794 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 80b3ba1d834a4314b63f336675845441 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441
I20240502 08:20:32.878159 25934 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:32.880031 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b3ba1d834a4314b63f336675845441. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.880594 21811 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.881258 25934 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.882117 21682 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:32.882720 21964 tablet_copy_source_session.cc:205] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:32.888952 25934 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:32.896008 26365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:32.901911 21811 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:32.904043 23871 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:20:32.904901 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20b2fd9f1ce488ba7793ed7ac268c16. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:32.905210 26075 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.908829 26365 tablet_copy_client.cc:783] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:32.909140 26365 tablet_copy_client.cc:647] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:32.911829 26365 tablet_copy_client.cc:523] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:32.916350 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:32.916621 21941 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.917937 21550 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:32.919052 25900 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.919970 25900 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:32.935966 21411 catalog_manager.cc:5484] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:32.941282 26075 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.944087 21550 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.947844 26115 ts_tablet_manager.cc:902] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:32.957120 23871 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.967105 21406 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:32.972183 21682 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:32.990729 26347 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:33.003692 21944 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:33.018157 26097 tablet_copy_client.cc:308] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:33.019467 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 59f5d97e6e0a4755be41699246907e55 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:55505)
I20240502 08:20:33.019728 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 59f5d97e6e0a4755be41699246907e55 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:55505: session id = 4b9bfd2541144dad86507451343640d3-59f5d97e6e0a4755be41699246907e55
I20240502 08:20:33.023927 21572 tablet_copy_source_session.cc:205] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:33.030078 26097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:33.041371 26115 tablet_copy_client.cc:308] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:33.042480 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet dc49c050280346db88d1323e30513c80 from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:35393)
I20240502 08:20:33.042673 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet dc49c050280346db88d1323e30513c80 from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:35393: session id = 95f32202b9db457fb09872f6f51ea8ba-dc49c050280346db88d1323e30513c80
I20240502 08:20:33.045734 21833 tablet_copy_source_session.cc:205] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:33.047389 26115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:33.049144 26097 tablet_copy_client.cc:783] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:33.049459 26097 tablet_copy_client.cc:647] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:33.052541 26097 tablet_copy_client.cc:523] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:33.055969 26115 tablet_copy_client.cc:783] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:33.056350 26115 tablet_copy_client.cc:647] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:33.058645 26115 tablet_copy_client.cc:523] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:33.069752 26115 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:33.079867 26097 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:33.165180 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:20:33.165831 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:33.166185 26365 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.250s user 0.069s sys 0.028s
I20240502 08:20:33.167163 26365 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:33.176554 26365 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.182408 26365 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:33.241519 26365 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.075s user 0.009s sys 0.011s
I20240502 08:20:33.253091 26115 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:33.253619 26115 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:33.253938 26115 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.186s user 0.039s sys 0.021s
I20240502 08:20:33.254776 26115 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:33.255082 26115 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.255424 26115 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:33.256093 26115 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:20:33.256956 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-dc49c050280346db88d1323e30513c80 received from {username='slave'} at 127.8.68.200:35393
I20240502 08:20:33.257108 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-dc49c050280346db88d1323e30513c80 on tablet dc49c050280346db88d1323e30513c80 with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:33.259341 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:20:33.259542 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441 on tablet 80b3ba1d834a4314b63f336675845441 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:33.297057 26453 ts_tablet_manager.cc:902] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:33.324736 26347 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:33.325330 26347 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:33.325691 26347 ts_tablet_manager.cc:1367] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.335s user 0.057s sys 0.007s
I20240502 08:20:33.326642 26347 raft_consensus.cc:348] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:33.326959 26347 raft_consensus.cc:729] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.327313 26347 consensus_queue.cc:260] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:33.361531 26347 ts_tablet_manager.cc:1398] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.036s user 0.010s sys 0.000s
I20240502 08:20:33.362684 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-59f5d97e6e0a4755be41699246907e55 received from {username='slave'} at 127.8.68.197:52703
I20240502 08:20:33.362844 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-59f5d97e6e0a4755be41699246907e55 on tablet 59f5d97e6e0a4755be41699246907e55 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:33.363765 26097 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:33.364233 26097 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:33.364549 26097 ts_tablet_manager.cc:1367] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.285s user 0.054s sys 0.004s
I20240502 08:20:33.380301 26097 raft_consensus.cc:348] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:33.380826 26097 raft_consensus.cc:729] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.382673 26097 consensus_queue.cc:260] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:33.387611 26453 tablet_copy_client.cc:308] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:33.388419 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 80b3ba1d834a4314b63f336675845441 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53343)
I20240502 08:20:33.388592 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 80b3ba1d834a4314b63f336675845441 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53343: session id = 20b637fbec5444dd91b3af9cdda482f5-80b3ba1d834a4314b63f336675845441
I20240502 08:20:33.391647 21964 tablet_copy_source_session.cc:205] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:33.393416 26453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:33.400394 26453 tablet_copy_client.cc:783] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:33.400797 26453 tablet_copy_client.cc:647] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:33.402887 26453 tablet_copy_client.cc:523] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:33.407631 26453 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:20:33.412842 26097 ts_tablet_manager.cc:1398] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.048s user 0.001s sys 0.005s
I20240502 08:20:33.414065 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-59f5d97e6e0a4755be41699246907e55 received from {username='slave'} at 127.8.68.198:55505
I20240502 08:20:33.414207 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 4b9bfd2541144dad86507451343640d3-59f5d97e6e0a4755be41699246907e55 on tablet 59f5d97e6e0a4755be41699246907e55 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:33.438539 23871 raft_consensus.cc:1203] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:33.447563 26365 tablet_replica.cc:330] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: stopping tablet replica
I20240502 08:20:33.447907 26365 raft_consensus.cc:2229] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:33.448089 26365 pending_rounds.cc:62] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Trying to abort 1 pending ops.
I20240502 08:20:33.448241 26365 pending_rounds.cc:68] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023157382635659264 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "80b3ba1d834a4314b63f336675845441" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } } }
I20240502 08:20:33.448731 26365 raft_consensus.cc:2894] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:20:33.448911 26365 raft_consensus.cc:2866] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:20:33.449024 26365 raft_consensus.cc:2258] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:33.450143 26365 ts_tablet_manager.cc:1875] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:33.457990 26365 ts_tablet_manager.cc:1888] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:33.460512 26365 log.cc:1196] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/wals/80b3ba1d834a4314b63f336675845441
I20240502 08:20:33.461603 26365 ts_tablet_manager.cc:902] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:20:33.489959 26365 tablet_copy_client.cc:308] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:20:33.490891 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 80b3ba1d834a4314b63f336675845441 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:20:33.491092 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 80b3ba1d834a4314b63f336675845441 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441
I20240502 08:20:33.495024 21964 tablet_copy_source_session.cc:205] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:33.496465 26365 ts_tablet_manager.cc:1875] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:20:33.506567 26365 ts_tablet_manager.cc:1888] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:20:33.506896 26365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b3ba1d834a4314b63f336675845441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:33.510138 26365 tablet_copy_client.cc:783] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:33.510392 26365 tablet_copy_client.cc:647] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:33.524263 26518 ts_tablet_manager.cc:902] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:33.526958 26365 tablet_copy_client.cc:523] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:33.530877 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:33.533769 26453 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:33.536899 26453 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:20:33.537266 26453 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.130s user 0.058s sys 0.004s
I20240502 08:20:33.538151 26453 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:33.538434 26453 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.541536 26453 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:33.542316 26453 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:20:33.543591 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-80b3ba1d834a4314b63f336675845441 received from {username='slave'} at 127.8.68.197:53343
I20240502 08:20:33.543735 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-80b3ba1d834a4314b63f336675845441 on tablet 80b3ba1d834a4314b63f336675845441 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:33.551920 23470 raft_consensus.cc:1203] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:33.560988 21811 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
W20240502 08:20:33.564759 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bfc3eaed583458b8381d4e7b4d36aef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:33.564956 21682 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:33.566156 26009 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:33.567407 21550 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:33.568351 26009 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:33.676929 26009 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:33.678179 21682 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:33.686846 21405 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:33.759469 21550 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:33.778040 21747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 51741.7us
I20240502 08:20:33.786793 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:20:33.778380 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 73431.6us
I20240502 08:20:33.823608 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:33.835347 26518 tablet_copy_client.cc:308] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:33.836079 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:33.836280 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-a20b2fd9f1ce488ba7793ed7ac268c16
I20240502 08:20:33.839730 21703 tablet_copy_source_session.cc:205] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:33.842360 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:33.868777 26518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:33.872030 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:33.894619 23470 raft_consensus.cc:1203] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:33.903693 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:33.921408 26518 tablet_copy_client.cc:783] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:33.921743 26518 tablet_copy_client.cc:647] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:33.939563 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:33.950421 26518 tablet_copy_client.cc:523] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:33.954326 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:33.954865 26365 tablet_bootstrap.cc:492] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:33.955174 26365 ts_tablet_manager.cc:1367] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.424s user 0.055s sys 0.004s
I20240502 08:20:33.956111 26365 raft_consensus.cc:348] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:33.956410 26365 raft_consensus.cc:729] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:33.958518 26518 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:33.972579 23602 raft_consensus.cc:1203] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:33.968173 26365 consensus_queue.cc:260] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:20:33.988473 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 51872.1us
I20240502 08:20:33.992653 26365 ts_tablet_manager.cc:1398] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.037s user 0.002s sys 0.004s
I20240502 08:20:33.993914 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:20:33.994069 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-80b3ba1d834a4314b63f336675845441 on tablet 80b3ba1d834a4314b63f336675845441 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:34.089677 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.089840 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.115197 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.117432 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.129323 26610 ts_tablet_manager.cc:902] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:34.138718 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.149868 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:34.151345 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.158670 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:34.169157 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.241643 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.304507 26610 tablet_copy_client.cc:308] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:20:34.305821 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 2bfc3eaed583458b8381d4e7b4d36aef from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:47771)
I20240502 08:20:34.306066 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 2bfc3eaed583458b8381d4e7b4d36aef from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:47771: session id = 4b9bfd2541144dad86507451343640d3-2bfc3eaed583458b8381d4e7b4d36aef
I20240502 08:20:34.309738 21833 tablet_copy_source_session.cc:205] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:34.311192 26610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfc3eaed583458b8381d4e7b4d36aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:34.334496 26610 tablet_copy_client.cc:783] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:34.337955 26610 tablet_copy_client.cc:647] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:34.342695 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:34.347784 26610 tablet_copy_client.cc:523] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:34.352317 26610 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:20:34.378510 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.378855 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.423527 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.431607 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.441608 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.460048 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:34.495898 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.578431 23735 raft_consensus.cc:1203] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:34.652419 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.700256 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:34.715849 26610 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:34.717404 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.717816 26610 tablet_bootstrap.cc:492] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:20:34.719524 26610 ts_tablet_manager.cc:1367] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.367s user 0.052s sys 0.015s
I20240502 08:20:34.720547 26610 raft_consensus.cc:348] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:34.721673 26610 raft_consensus.cc:729] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:20:34.722002 26610 consensus_queue.cc:260] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:34.727203 26518 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:34.727893 26518 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:34.729770 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.739789 26518 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.781s user 0.055s sys 0.004s
I20240502 08:20:34.740715 26518 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:34.741050 26518 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:34.747908 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:34.748191 26518 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:34.748939 26518 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:20:34.750841 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-a20b2fd9f1ce488ba7793ed7ac268c16 received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:34.750995 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-a20b2fd9f1ce488ba7793ed7ac268c16 on tablet a20b2fd9f1ce488ba7793ed7ac268c16 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:34.779052 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:34.786659 26610 ts_tablet_manager.cc:1398] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20240502 08:20:34.795583 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-2bfc3eaed583458b8381d4e7b4d36aef received from {username='slave'} at 127.8.68.198:47771
I20240502 08:20:34.795778 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 4b9bfd2541144dad86507451343640d3-2bfc3eaed583458b8381d4e7b4d36aef on tablet 2bfc3eaed583458b8381d4e7b4d36aef with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:20:34.806630 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.814749 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:34.978410 23603 raft_consensus.cc:1203] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:34.993986 23735 raft_consensus.cc:1203] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:20:35.303197 21683 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:35.309149 21811 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:35.309442 21944 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:20:35.309878 26075 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:20:35.311923 25900 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:20:35.312916 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d933eacd944d84b39f0cb9c45641f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:35.374439 25900 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:35.386698 21811 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:35.393397 21944 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:35.399729 21408 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index -1 to 4, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:35.500586 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:20:35.546963 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 59973.6us
I20240502 08:20:35.584553 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:35.603091 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:35.608995 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:35.616621 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:35.670073 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:35.697019 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:35.730893 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:35.878899 26850 ts_tablet_manager.cc:902] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:20:35.981681 26850 tablet_copy_client.cc:308] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:20:35.983105 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d4d933eacd944d84b39f0cb9c45641f8 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:40593)
I20240502 08:20:35.983407 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d4d933eacd944d84b39f0cb9c45641f8 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:40593: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:20:36.005825 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:36.007411 21703 tablet_copy_source_session.cc:205] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:36.009660 26850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:36.043599 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.044750 26850 tablet_copy_client.cc:783] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:36.045364 26850 tablet_copy_client.cc:647] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:36.053982 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.055722 26850 tablet_copy_client.cc:523] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:36.058929 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.073098 26850 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:20:36.117571 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:36.130062 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.157909 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.209797 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.269522 21683 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:36.320019 23603 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:36.322535 26604 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:36.324903 21943 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:36.328979 26604 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:36.364491 26075 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:36.369215 21944 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:36.377521 23603 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:36.386930 21409 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:36.418350 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.434041 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:36.437028 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.516537 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.518468 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:36.549167 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet b88c4a1a36424086aad89b7585c17960 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:36.586470 26932 tablet_replica.cc:330] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:36.587302 26932 raft_consensus.cc:2229] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:36.587988 26932 raft_consensus.cc:2258] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:36.609144 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.610641 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.613448 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.691720 26850 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:36.693394 26850 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:20:36.693745 26850 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.621s user 0.055s sys 0.009s
I20240502 08:20:36.694578 26850 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:36.678674 26932 ts_tablet_manager.cc:1875] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:36.694821 26850 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:36.720455 26850 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:36.736861 26850 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.043s user 0.002s sys 0.001s
I20240502 08:20:36.738664 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-d4d933eacd944d84b39f0cb9c45641f8 received from {username='slave'} at 127.8.68.199:40593
I20240502 08:20:36.738826 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-d4d933eacd944d84b39f0cb9c45641f8 on tablet d4d933eacd944d84b39f0cb9c45641f8 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:36.777244 26932 ts_tablet_manager.cc:1888] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:36.780006 26932 log.cc:1196] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/b88c4a1a36424086aad89b7585c17960
I20240502 08:20:36.801054 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet b88c4a1a36424086aad89b7585c17960 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
I20240502 08:20:36.846374 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.868789 23735 raft_consensus.cc:1203] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:20:36.912204 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.918680 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:36.918956 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:36.977978 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.979483 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:36.985946 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:36.997041 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:37.077788 21811 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:37.110350 23735 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.110832 21944 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.111037 23871 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.111528 26928 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.113881 26555 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.114745 26555 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.147905 23870 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:37.146838 26928 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:37.163301 21944 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:37.167953 21405 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:37.172094 23735 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:37.236253 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de41d6c6de0400397645c05a0fa17de with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:37.241797 27018 tablet_replica.cc:330] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:37.242455 27018 raft_consensus.cc:2229] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:37.242990 27018 raft_consensus.cc:2258] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:37.244947 27018 ts_tablet_manager.cc:1875] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:37.266925 27018 ts_tablet_manager.cc:1888] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:37.303050 27018 log.cc:1196] T 2de41d6c6de0400397645c05a0fa17de P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/2de41d6c6de0400397645c05a0fa17de
I20240502 08:20:37.308710 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 2de41d6c6de0400397645c05a0fa17de (table rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]) successfully deleted
I20240502 08:20:37.328003 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:37.362114 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:37.365211 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:37.386348 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:37.395360 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:37.413898 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:37.470388 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:37.476864 21683 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:37.480242 23737 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.480674 21944 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.483297 25900 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.494849 26075 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.515398 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:37.534726 23603 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:37.535679 25900 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:37.575932 25900 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:37.588068 21944 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:37.595105 21409 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:37.597087 23603 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:37.599992 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 182a6f2df91242cfa10f05ef788a19cb with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:37.600556 27018 tablet_replica.cc:330] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:37.600813 27018 raft_consensus.cc:2229] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:37.601347 27018 raft_consensus.cc:2258] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:37.602190 27018 ts_tablet_manager.cc:1875] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:37.617420 27018 ts_tablet_manager.cc:1888] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:37.618193 23737 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:37.619313 27018 log.cc:1196] T 182a6f2df91242cfa10f05ef788a19cb P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/182a6f2df91242cfa10f05ef788a19cb
I20240502 08:20:37.620453 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 182a6f2df91242cfa10f05ef788a19cb (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:20:38.037896 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:38.087744 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.089182 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.096252 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:38.126649 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.198225 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.258810 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.306931 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.359503 21682 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:38.377671 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.392992 23737 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:38.393721 23870 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:38.394882 25900 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:38.395983 26075 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:38.400394 21944 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:38.401371 25900 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:38.404786 25900 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:38.405941 23737 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:38.412918 21944 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:38.427544 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:38.431548 21402 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:38.454206 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.460376 23870 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:38.484195 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:38.491891 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.557273 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.631904 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet a20b2fd9f1ce488ba7793ed7ac268c16 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:38.641548 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.674134 27204 tablet_replica.cc:330] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:38.674396 27204 raft_consensus.cc:2229] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:38.674661 27204 raft_consensus.cc:2258] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:38.675485 27204 ts_tablet_manager.cc:1875] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:38.680891 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.764190 27204 ts_tablet_manager.cc:1888] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:38.764477 27204 log.cc:1196] T a20b2fd9f1ce488ba7793ed7ac268c16 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/a20b2fd9f1ce488ba7793ed7ac268c16
I20240502 08:20:38.769647 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet a20b2fd9f1ce488ba7793ed7ac268c16 (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:20:38.820109 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.832392 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:38.884904 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.907970 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:38.915551 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:38.925621 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:38.942041 21747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 72630.2us
I20240502 08:20:38.979840 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:38.989805 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.155462 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:39.177604 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.208492 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:39.215234 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:39.249202 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.252977 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.307984 21551 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:39.341610 21682 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:39.342435 27060 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:39.342751 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.357993 23602 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:39.374419 27288 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.375624 27289 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:39.380251 21683 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.387831 21407 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:39.429136 21486 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 69318.6us
I20240502 08:20:39.439734 23602 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.495514 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:39.500013 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:39.540211 27320 tablet_replica.cc:330] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:39.543426 27320 raft_consensus.cc:2229] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:39.547261 27320 raft_consensus.cc:2258] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:39.568576 27320 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:39.573370 21682 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:39.580272 23602 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:39.581333 25900 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:39.580550 21943 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:39.597355 25900 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:39.599741 27180 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.600854 23603 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.608738 21402 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.621215 21944 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:39.624388 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a45b1da348e4bb89f021df103a0603f with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:39.650768 27331 tablet_replica.cc:330] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:39.658720 27331 raft_consensus.cc:2229] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:39.659163 27331 raft_consensus.cc:2258] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:39.661322 27331 ts_tablet_manager.cc:1875] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:39.673633 27331 ts_tablet_manager.cc:1888] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:39.679394 27331 log.cc:1196] T 8a45b1da348e4bb89f021df103a0603f P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a45b1da348e4bb89f021df103a0603f
I20240502 08:20:39.680990 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 8a45b1da348e4bb89f021df103a0603f (table rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]) successfully deleted
I20240502 08:20:39.716495 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.716683 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.743518 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:39.747944 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.760219 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:39.800714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:39.813256 27320 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:39.813482 27320 log.cc:1196] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:20:39.818707 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 9c0a886b3ba04737a6988c3b58c9698e (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
W20240502 08:20:39.839063 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 52909.5us
I20240502 08:20:39.849958 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:39.873803 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:39.886502 21682 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:39.912472 23735 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:39.913252 27362 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:39.921248 23870 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:39.922096 27362 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:39.944125 21551 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:39.944963 27362 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:39.959960 27362 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:39.972525 21550 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:39.972872 23871 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:39.980526 21408 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:39.983693 23471 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:39.987058 27180 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:39.991160 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:39.991652 27320 tablet_replica.cc:330] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:39.991971 27320 raft_consensus.cc:2229] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:39.992373 27320 raft_consensus.cc:2258] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:39.993271 27320 ts_tablet_manager.cc:1875] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:40.000576 23735 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.014501 27320 ts_tablet_manager.cc:1888] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:40.017874 27320 log.cc:1196] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:20:40.020038 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:20:40.032828 23470 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.066316 21551 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:40.110963 23602 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.111925 27289 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.121572 23470 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.122529 27289 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.128818 21944 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.130318 27289 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:40.147086 27409 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.165283 21943 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:40.177927 21405 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:40.182502 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f5d97e6e0a4755be41699246907e55 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:40.216737 27419 tablet_replica.cc:330] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:40.221648 23602 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:40.229837 23471 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:40.253324 27419 raft_consensus.cc:2229] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:40.254091 27419 raft_consensus.cc:2258] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:40.255887 27419 ts_tablet_manager.cc:1875] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:40.280579 27419 ts_tablet_manager.cc:1888] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:40.284961 27419 log.cc:1196] T 59f5d97e6e0a4755be41699246907e55 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/59f5d97e6e0a4755be41699246907e55
I20240502 08:20:40.302287 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 59f5d97e6e0a4755be41699246907e55 (table rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]) successfully deleted
I20240502 08:20:40.369815 21551 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:40.378304 23735 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.379573 23602 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.379814 27289 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.380353 27410 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.383420 21812 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:40.384088 27289 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:40.408829 27289 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.415335 21812 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.417642 23602 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.443977 23735 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.448973 21402 catalog_manager.cc:5484] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:40.458048 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet e0241228bd204ca2a8ad2c57b6eeb543 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:40.494922 27476 tablet_replica.cc:330] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:40.495676 27476 raft_consensus.cc:2229] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:40.496105 27476 raft_consensus.cc:2258] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:40.497774 27476 ts_tablet_manager.cc:1875] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:40.532079 27476 ts_tablet_manager.cc:1888] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:40.547778 27476 log.cc:1196] T e0241228bd204ca2a8ad2c57b6eeb543 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0241228bd204ca2a8ad2c57b6eeb543
I20240502 08:20:40.549507 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet e0241228bd204ca2a8ad2c57b6eeb543 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
I20240502 08:20:40.777186 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:40.809629 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:40.819947 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:40.857281 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:40.857959 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:40.866042 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:40.872735 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:40.876312 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:40.899217 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:40.909606 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:40.949227 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:40.988554 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:40.989197 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.007774 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.031355 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.043816 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:41.066615 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.078650 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.111694 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:41.123205 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:41.144528 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:41.144994 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.169715 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.192034 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:41.500276 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.501457 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.542393 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:41.556908 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:41.564203 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:41.584383 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.607522 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.626793 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:41.871579 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:41.916141 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:41.928481 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.930119 21811 consensus_queue.cc:237] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:41.933357 21812 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:41.939529 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:41.948783 23871 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:41.950428 26555 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:41.951009 21550 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:41.951756 27480 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:41.976398 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.981252 27629 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:41.985445 21551 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:41.985546 23870 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:41.990834 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:41.991642 21408 catalog_manager.cc:5484] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:42.011732 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:42.011850 23602 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.012600 26555 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.024144 21550 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.024932 26555 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.057376 27480 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:42.063841 21551 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:42.071100 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:42.094553 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 53212.9us
I20240502 08:20:42.123497 23602 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:42.125919 21409 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:42.129200 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet c4862ea2f67c43a28689b464d8ec96e6 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:42.139804 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfc3eaed583458b8381d4e7b4d36aef with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:42.170164 27715 tablet_replica.cc:330] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:42.170614 27715 raft_consensus.cc:2229] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:42.171007 27715 raft_consensus.cc:2258] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:42.172215 27715 ts_tablet_manager.cc:1875] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:42.174420 27711 tablet_replica.cc:330] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:42.174789 27711 raft_consensus.cc:2229] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:42.175227 27711 raft_consensus.cc:2258] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:42.176081 27711 ts_tablet_manager.cc:1875] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:42.207967 21943 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:42.212045 23737 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:42.212045 21811 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:42.213183 27715 ts_tablet_manager.cc:1888] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:42.213356 27715 log.cc:1196] T 2bfc3eaed583458b8381d4e7b4d36aef P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/2bfc3eaed583458b8381d4e7b4d36aef
I20240502 08:20:42.214251 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 2bfc3eaed583458b8381d4e7b4d36aef (table rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]) successfully deleted
I20240502 08:20:42.223874 23469 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:42.223903 27479 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 08:20:42.225579 27661 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 08:20:42.226779 27479 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:42.259735 27711 ts_tablet_manager.cc:1888] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:42.271260 21811 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.279589 23737 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.280736 21405 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.287082 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:42.287564 27715 tablet_replica.cc:330] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:42.287868 27715 raft_consensus.cc:2229] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:42.288151 27715 raft_consensus.cc:2258] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:42.289116 27715 ts_tablet_manager.cc:1875] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:42.297010 27715 ts_tablet_manager.cc:1888] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:42.297178 27715 log.cc:1196] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:20:42.298455 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
I20240502 08:20:42.269992 27726 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.304394 27711 log.cc:1196] T c4862ea2f67c43a28689b464d8ec96e6 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/c4862ea2f67c43a28689b464d8ec96e6
I20240502 08:20:42.311640 23469 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.316609 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet c4862ea2f67c43a28689b464d8ec96e6 (table rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]) successfully deleted
I20240502 08:20:42.350729 21683 consensus_queue.cc:237] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:42.375515 23469 raft_consensus.cc:1261] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.377879 27362 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.382608 21943 raft_consensus.cc:1261] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.384836 27692 consensus_queue.cc:1035] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.408533 27362 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.412289 21944 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.417389 21405 catalog_manager.cc:5484] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.423658 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet efec2cc693bd4dd992d0bec22fca5419 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:42.424829 23471 raft_consensus.cc:2932] T efec2cc693bd4dd992d0bec22fca5419 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:42.448266 27767 tablet_replica.cc:330] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:42.452153 27767 raft_consensus.cc:2229] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:42.454046 27767 raft_consensus.cc:2258] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:42.457267 27767 ts_tablet_manager.cc:1875] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:42.502813 27767 ts_tablet_manager.cc:1888] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:42.502980 27767 log.cc:1196] T efec2cc693bd4dd992d0bec22fca5419 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/efec2cc693bd4dd992d0bec22fca5419
I20240502 08:20:42.505455 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet efec2cc693bd4dd992d0bec22fca5419 (table rebalance_test_table_7 [id=7dfe697bbc4946eebea2280bb24e7506]) successfully deleted
I20240502 08:20:42.619416 21811 consensus_queue.cc:237] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:42.625669 23737 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.626384 21683 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:42.626921 27629 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.627372 27693 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:42.633340 27629 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:42.636070 23737 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:42.646865 21683 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:42.655754 21409 catalog_manager.cc:5484] T 7b3f6a0b2561419ca6bd832fd65e3196 P fd5e97ae7e344e4687b15ca0cde8d0ba reported cstate change: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:42.666507 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3f6a0b2561419ca6bd832fd65e3196 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:42.690955 27825 tablet_replica.cc:330] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:42.691380 27825 raft_consensus.cc:2229] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:42.691821 27825 raft_consensus.cc:2258] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:42.740199 27825 ts_tablet_manager.cc:1875] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:42.758670 27825 ts_tablet_manager.cc:1888] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:42.758867 27825 log.cc:1196] T 7b3f6a0b2561419ca6bd832fd65e3196 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/7b3f6a0b2561419ca6bd832fd65e3196
I20240502 08:20:42.771226 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 7b3f6a0b2561419ca6bd832fd65e3196 (table rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]) successfully deleted
I20240502 08:20:42.886729 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:42.920547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:42.942874 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:42.950379 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:42.991837 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.027086 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.069566 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.120206 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.156780 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.208127 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.231859 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.233198 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:43.273689 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:43.276758 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.334317 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.349769 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.613775 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.682688 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:43.692682 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.713915 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.746281 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:43.754866 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.795346 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.847373 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.864033 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:43.870668 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.887022 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:43.890178 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.908227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.925432 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:43.936640 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:43.958752 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.174825 21683 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:44.224254 23602 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:44.224254 23737 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:44.225114 25900 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:44.226250 25900 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:44.241058 27362 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:44.246110 23735 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:44.250923 21407 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:44.283082 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 182a6f2df91242cfa10f05ef788a19cb with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:44.283968 28021 tablet_replica.cc:330] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:44.284286 28021 raft_consensus.cc:2229] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:44.284646 28021 raft_consensus.cc:2258] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:44.285921 28021 ts_tablet_manager.cc:1875] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:44.294108 28021 ts_tablet_manager.cc:1888] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:44.294415 28021 log.cc:1196] T 182a6f2df91242cfa10f05ef788a19cb P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/182a6f2df91242cfa10f05ef788a19cb
I20240502 08:20:44.295552 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 182a6f2df91242cfa10f05ef788a19cb (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:20:44.299484 23602 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:44.395140 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:44.424875 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:44.471707 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.476580 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.490396 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:44.529911 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.532522 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:44.585028 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.714406 21550 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:44.743669 23602 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:44.744454 27289 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:44.817091 27289 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:44.820201 23601 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:44.825670 21403 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:44.925734 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:44.926415 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:44.957439 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:44.966820 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:44.988154 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.029557 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:45.053182 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.121320 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:45.122799 21550 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:45.126313 28161 tablet_replica.cc:330] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:45.126474 23601 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:45.126551 28161 raft_consensus.cc:2229] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:45.126752 28161 raft_consensus.cc:2258] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:45.127547 28161 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:45.130052 28038 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:20:45.135641 21488 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:45.142238 28038 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:45.153569 23601 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:45.168224 28161 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:45.168393 28161 log.cc:1196] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:20:45.170418 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 9c0a886b3ba04737a6988c3b58c9698e (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
I20240502 08:20:45.181520 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9c0a886b3ba04737a6988c3b58c9698e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:20:45.196277 21409 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:20:45.202466 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.215785 21811 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:45.319355 23735 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:45.320524 21550 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:45.321413 27629 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:45.322729 27629 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:45.333180 23871 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:45.333897 28164 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:45.337807 27629 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.339226 23871 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.345878 23735 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.348559 21409 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.349025 21550 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.354192 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet dc49c050280346db88d1323e30513c80 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:45.383623 28189 tablet_replica.cc:330] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:45.383936 28189 raft_consensus.cc:2229] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:45.384342 28189 raft_consensus.cc:2258] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:45.385373 28189 ts_tablet_manager.cc:1875] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:45.403641 28189 ts_tablet_manager.cc:1888] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:45.403909 28189 log.cc:1196] T dc49c050280346db88d1323e30513c80 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/dc49c050280346db88d1323e30513c80
I20240502 08:20:45.404731 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet dc49c050280346db88d1323e30513c80 (table rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]) successfully deleted
I20240502 08:20:45.707659 28256 ts_tablet_manager.cc:902] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:45.711905 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:45.721568 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:45.743335 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.756232 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.764336 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:45.792454 21682 consensus_queue.cc:237] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:45.805999 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.810536 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.830703 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:45.862192 23871 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:45.863039 28000 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:45.872011 21944 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:45.872773 28000 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:45.874429 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:45.885059 28272 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.885977 23870 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.892375 21409 catalog_manager.cc:5484] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.904093 28256 tablet_copy_client.cc:308] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:45.907196 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:52239)
I20240502 08:20:45.907451 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:52239: session id = 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:20:45.910645 21572 tablet_copy_source_session.cc:205] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:45.921859 21943 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:45.922811 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.930035 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:45.933362 28256 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:20:45.946233 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:45.995457 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 1648586b889b4a1e9ac34481cec6257a with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:46.017280 28295 tablet_replica.cc:330] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:46.017627 28295 raft_consensus.cc:2229] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:46.017968 28295 raft_consensus.cc:2258] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:46.018885 28295 ts_tablet_manager.cc:1875] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:46.031987 28256 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:20:46.032862 28256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:46.058377 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:46.058881 23451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.060909 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.062109 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:46.076583 28256 tablet_copy_client.cc:783] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:46.079775 28256 tablet_copy_client.cc:647] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:46.088959 28256 tablet_copy_client.cc:523] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:46.094095 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.096705 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:46.106359 28256 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:20:46.110236 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:46.125020 28295 ts_tablet_manager.cc:1888] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:46.125224 28295 log.cc:1196] T 1648586b889b4a1e9ac34481cec6257a P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/1648586b889b4a1e9ac34481cec6257a
I20240502 08:20:46.127370 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 1648586b889b4a1e9ac34481cec6257a (table rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]) successfully deleted
I20240502 08:20:46.134507 21662 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:20:46.183710 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.203009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:46.206094 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.294018 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.473263 28256 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:46.473798 28256 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:20:46.474107 28256 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.368s user 0.059s sys 0.003s
I20240502 08:20:46.474929 28256 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:46.475149 28256 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:20:46.475438 28256 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:20:46.475991 28256 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:20:46.477928 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e received from {username='slave'} at 127.8.68.196:52239
I20240502 08:20:46.478124 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e on tablet 9c0a886b3ba04737a6988c3b58c9698e with peer 0f655ab8f20d4ba482c3173f9594b98b
I20240502 08:20:46.548676 21682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9"
dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48282
I20240502 08:20:46.548965 21682 raft_consensus.cc:595] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:20:46.555193 28000 raft_consensus.cc:979] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: : Instructing follower 20b637fbec5444dd91b3af9cdda482f5 to start an election
I20240502 08:20:46.555423 28000 raft_consensus.cc:1067] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Signalling peer 20b637fbec5444dd91b3af9cdda482f5 to start an election
I20240502 08:20:46.561707 23471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9"
dest_uuid: "20b637fbec5444dd91b3af9cdda482f5"
from {username='slave'} at 127.8.68.194:45173
I20240502 08:20:46.562106 23471 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:20:46.562355 23471 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:46.565847 23471 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:46.579371 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:20:46.579672 21683 raft_consensus.cc:3032] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:20:46.579778 21683 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:20:46.580194 21683 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:46.580796 21683 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:46.584723 21683 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:46.592200 21811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:46.592509 21811 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:46.595928 21811 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:46.596580 23406 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 20b637fbec5444dd91b3af9cdda482f5, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:20:46.603523 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:46.603786 21550 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:46.607199 21550 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:46.609019 23471 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 2 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851), cfff665685a14d8796aece26addab59c (127.8.68.193:46845), fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397)
I20240502 08:20:46.615017 28343 raft_consensus.cc:2781] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:20:46.646649 28343 raft_consensus.cc:686] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Becoming Leader. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Running, Role: LEADER
I20240502 08:20:46.647622 28343 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:46.651229 23735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
I20240502 08:20:46.651698 23735 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:46.663012 23735 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:46.680550 21403 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: term changed from 1 to 2, leader changed from 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) to 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:20:46.753718 21747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 43482.3us
I20240502 08:20:46.791054 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:46.815585 21944 raft_consensus.cc:1203] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:20:46.821638 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:46.822990 23451 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:20:46.847229 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:46.849632 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.923359 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:46.927750 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:46.963491 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.082331 23735 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:47.083407 28343 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.130210 21683 consensus_queue.cc:237] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:47.131121 21811 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:47.134631 28343 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.168891 23471 raft_consensus.cc:1261] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:47.170058 27362 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:47.178122 21549 raft_consensus.cc:1261] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:47.179085 27362 consensus_queue.cc:1035] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:47.182181 28272 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:47.193274 23469 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:47.202862 21403 catalog_manager.cc:5484] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:47.228904 21552 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:47.229851 28343 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.233965 21549 raft_consensus.cc:2932] T 3898bfdb9c3d4a05af78f79fd2de61e2 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:47.251967 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 3898bfdb9c3d4a05af78f79fd2de61e2 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:47.260986 21683 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:47.262135 28445 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.322650 28450 tablet_replica.cc:330] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:47.328696 28450 raft_consensus.cc:2229] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:47.329061 28450 raft_consensus.cc:2258] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:47.329931 28450 ts_tablet_manager.cc:1875] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:47.340755 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:47.353653 21683 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:47.364403 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:47.378422 23451 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:20:47.386602 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.394263 28450 ts_tablet_manager.cc:1888] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:47.394407 28450 log.cc:1196] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/3898bfdb9c3d4a05af78f79fd2de61e2
I20240502 08:20:47.415750 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 3898bfdb9c3d4a05af78f79fd2de61e2 (table rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]) successfully deleted
I20240502 08:20:47.434648 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:47.478850 23735 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:47.479621 28441 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.486626 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:47.488282 23471 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:47.489637 21811 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:47.490355 28441 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.491056 28272 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:47.540640 28272 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:47.545289 23469 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:47.549340 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.552395 21406 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:47.557607 23737 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:47.561661 21811 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:47.570789 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.583778 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 599d5781f1e24ae9ad425a34d88c6b68 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:47.601608 28492 tablet_replica.cc:330] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:47.601838 28492 raft_consensus.cc:2229] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:47.602141 28492 raft_consensus.cc:2258] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:47.602988 28492 ts_tablet_manager.cc:1875] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:47.616295 28492 ts_tablet_manager.cc:1888] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:47.633378 28492 log.cc:1196] T 599d5781f1e24ae9ad425a34d88c6b68 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/599d5781f1e24ae9ad425a34d88c6b68
I20240502 08:20:47.634683 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 599d5781f1e24ae9ad425a34d88c6b68 (table rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]) successfully deleted
I20240502 08:20:47.712468 21549 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:47.715332 23601 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:47.716215 28038 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:47.717011 23471 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:47.719393 28038 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:47.727216 28424 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:47.728091 23471 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:47.734990 21409 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:47.741719 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f5d97e6e0a4755be41699246907e55 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:47.753396 23602 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:47.766324 28520 tablet_replica.cc:330] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:47.767470 28520 raft_consensus.cc:2229] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:47.768110 28520 raft_consensus.cc:2258] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:47.769593 28520 ts_tablet_manager.cc:1875] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:47.803366 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:47.804066 28520 ts_tablet_manager.cc:1888] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:47.804250 28520 log.cc:1196] T 59f5d97e6e0a4755be41699246907e55 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/59f5d97e6e0a4755be41699246907e55
I20240502 08:20:47.806948 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 59f5d97e6e0a4755be41699246907e55 (table rebalance_test_table_4 [id=7fcdc1cbce1641c3b983eb474059b2c5]) successfully deleted
I20240502 08:20:47.822816 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:20:47.845786 21811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "203e7914da3f4494b239c6603513caee"
dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36608
I20240502 08:20:47.846113 21811 raft_consensus.cc:595] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:20:47.926147 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.927978 23451 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:20:47.961263 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.974957 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:47.981803 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:48.028146 28369 raft_consensus.cc:979] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: : Instructing follower 20b637fbec5444dd91b3af9cdda482f5 to start an election
I20240502 08:20:48.028375 28370 raft_consensus.cc:1067] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Signalling peer 20b637fbec5444dd91b3af9cdda482f5 to start an election
I20240502 08:20:48.038064 23471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "203e7914da3f4494b239c6603513caee"
dest_uuid: "20b637fbec5444dd91b3af9cdda482f5"
from {username='slave'} at 127.8.68.195:45899
I20240502 08:20:48.038383 23471 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:20:48.038506 23471 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:48.059720 23471 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:48.067936 21944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0f655ab8f20d4ba482c3173f9594b98b"
I20240502 08:20:48.068209 21944 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:48.072856 21811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:20:48.073155 21811 raft_consensus.cc:3032] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:20:48.073230 21811 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:20:48.073583 21811 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:48.074127 21811 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:48.082180 21944 raft_consensus.cc:2454] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:48.084419 21811 raft_consensus.cc:2454] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:48.085142 23406 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0f655ab8f20d4ba482c3173f9594b98b, 13e764b27ed449c09aee719cdb2597f8, 20b637fbec5444dd91b3af9cdda482f5; no voters:
I20240502 08:20:48.085615 28455 raft_consensus.cc:2781] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:20:48.085770 28455 raft_consensus.cc:686] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Becoming Leader. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Running, Role: LEADER
I20240502 08:20:48.086256 28455 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:48.091465 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e7914da3f4494b239c6603513caee" candidate_uuid: "20b637fbec5444dd91b3af9cdda482f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cfff665685a14d8796aece26addab59c"
I20240502 08:20:48.091861 21549 raft_consensus.cc:3037] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:20:48.094969 23471 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 2 election: Requested vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:20:48.104239 21549 raft_consensus.cc:2454] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b637fbec5444dd91b3af9cdda482f5 in term 2.
I20240502 08:20:48.130573 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:48.146390 21406 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: term changed from 1 to 2, leader changed from 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) to 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:48.406419 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:48.428380 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:48.434054 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:48.458443 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:48.460758 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:48.463779 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:48.475224 21549 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:20:48.476173 28455 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:48.488209 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:48.539736 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:48.710661 21811 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:20:48.711694 28455 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:48.714932 21944 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:20:48.716243 28455 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:48.775383 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 54332.4us
I20240502 08:20:48.909200 23471 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:48.931541 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:48.947983 23735 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:20:48.950539 21549 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:20:48.951148 21811 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:20:48.951143 28455 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:48.953374 28667 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:48.956704 28667 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:48.961683 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:48.975544 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:48.982957 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:49.028117 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:49.085901 28455 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:49.086793 23737 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:49.096316 21549 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:49.106079 21409 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 5 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:49.107220 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.157501 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:49.168943 21812 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:49.228682 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.258241 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.268994 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c17796e5d04014aafdac46984f4ba9 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:49.296386 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.296650 28772 tablet_replica.cc:330] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:49.296861 28772 raft_consensus.cc:2229] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:49.297044 28772 raft_consensus.cc:2258] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:49.297833 28772 ts_tablet_manager.cc:1875] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:49.305577 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:49.301771 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:49.308113 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.309877 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:49.317647 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:49.334580 28772 ts_tablet_manager.cc:1888] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:20:49.335260 28772 log.cc:1196] T 12c17796e5d04014aafdac46984f4ba9 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/12c17796e5d04014aafdac46984f4ba9
I20240502 08:20:49.347960 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:49.359611 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.360913 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 12c17796e5d04014aafdac46984f4ba9 (table rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]) successfully deleted
I20240502 08:20:49.363941 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:49.403633 21812 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:49.430183 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:49.441772 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:49.457459 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.488271 23602 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:49.489104 28753 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:49.496276 21549 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:49.498453 28753 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:49.501243 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:49.546590 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.570250 23601 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:49.579196 28752 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:49.590358 21405 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:49.594600 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 852613f14b324b38b5eecc29d4c602cd with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:49.612134 21549 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:49.620373 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:49.621819 28799 tablet_replica.cc:330] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:49.636924 28799 raft_consensus.cc:2229] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:49.637284 28799 raft_consensus.cc:2258] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:49.638070 28799 ts_tablet_manager.cc:1875] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:20:49.649521 21485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 59036.1us
I20240502 08:20:49.780787 28799 ts_tablet_manager.cc:1888] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:49.781029 28799 log.cc:1196] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/852613f14b324b38b5eecc29d4c602cd
I20240502 08:20:49.781929 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 852613f14b324b38b5eecc29d4c602cd (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:20:49.963559 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:49.989171 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:50.086310 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:50.124315 21812 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:50.127710 23871 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:50.131516 28753 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:50.141409 23737 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:50.142529 28164 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:50.157411 28753 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:50.162343 23871 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:50.170365 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:50.174553 21403 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:50.204685 23735 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:50.216171 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:50.262367 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:50.273988 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:50.289856 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet dc49c050280346db88d1323e30513c80 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:50.301826 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:50.315375 23471 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:50.332880 28910 tablet_replica.cc:330] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:50.336040 21943 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:50.336977 28455 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:50.345842 21549 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:20:50.353926 28910 raft_consensus.cc:2229] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:50.354233 28455 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:50.361348 21812 consensus_queue.cc:237] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:50.373328 23601 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:50.374142 28753 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:50.378536 28910 raft_consensus.cc:2258] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:50.387073 28910 ts_tablet_manager.cc:1875] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:20:50.397751 23407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 43949.4us
I20240502 08:20:50.400053 23469 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:50.400769 28753 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:50.396430 28927 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240502 08:20:50.403569 21943 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:50.404799 21403 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 4 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:50.409430 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e7914da3f4494b239c6603513caee with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:50.423187 28934 tablet_replica.cc:330] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:50.423733 28934 raft_consensus.cc:2229] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:50.424649 28934 raft_consensus.cc:2258] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:50.426093 28934 ts_tablet_manager.cc:1875] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:50.431761 21943 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:50.433351 28753 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:50.449180 28934 ts_tablet_manager.cc:1888] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:20:50.452522 28934 log.cc:1196] T 203e7914da3f4494b239c6603513caee P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/203e7914da3f4494b239c6603513caee
I20240502 08:20:50.453748 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 203e7914da3f4494b239c6603513caee (table rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]) successfully deleted
W20240502 08:20:50.451503 28926 log.cc:924] Time spent T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240502 08:20:50.453557 28753 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:50.454789 21943 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:50.456126 28910 ts_tablet_manager.cc:1888] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:50.456243 28910 log.cc:1196] T dc49c050280346db88d1323e30513c80 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc49c050280346db88d1323e30513c80
I20240502 08:20:50.471415 23602 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:50.469990 21403 catalog_manager.cc:5484] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:50.485385 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet d8042f6ab6404c5a9f8cce0b8f971542 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:50.497563 23469 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:50.466439 21549 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:50.514211 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet dc49c050280346db88d1323e30513c80 (table rebalance_test_table_1 [id=ec7f747ed54e4636a94dd2cd32110cfb]) successfully deleted
I20240502 08:20:50.542271 28944 tablet_replica.cc:330] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:50.542694 28944 raft_consensus.cc:2229] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:50.551788 28944 raft_consensus.cc:2258] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:50.552757 28944 ts_tablet_manager.cc:1875] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:50.681253 28944 ts_tablet_manager.cc:1888] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:50.681573 28944 log.cc:1196] T d8042f6ab6404c5a9f8cce0b8f971542 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/d8042f6ab6404c5a9f8cce0b8f971542
I20240502 08:20:50.683398 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet d8042f6ab6404c5a9f8cce0b8f971542 (table rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]) successfully deleted
I20240502 08:20:50.743490 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:50.793121 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:50.794867 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:50.832170 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:50.870548 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:50.878466 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:50.955619 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:50.988216 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:51.132565 21943 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:51.148555 23469 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:51.149232 28808 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:51.156401 21682 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:51.157166 28808 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:51.182720 28808 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.190982 21682 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.199631 21405 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.207147 23471 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.247057 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39750
I20240502 08:20:51.253626 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:51.262692 29077 tablet_replica.cc:330] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:20:51.263581 29077 raft_consensus.cc:2229] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:51.263942 29077 raft_consensus.cc:2258] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:51.265290 29077 ts_tablet_manager.cc:1875] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:51.281038 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:51.311406 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:51.321326 29077 ts_tablet_manager.cc:1888] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:51.321506 29077 log.cc:1196] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/76bb1e3e2b9d4e31bf93ac1ce08a7833
I20240502 08:20:51.322492 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:20:51.335983 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:51.357462 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:51.384359 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:51.396215 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:51.404356 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:51.721128 23469 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:51.734133 23737 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:20:51.735374 28925 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:20:51.742174 21549 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:20:51.743084 28925 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:20:51.791482 29146 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.797219 21549 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.809234 21405 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 7 to 8, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.848205 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:51.849733 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:51.892164 23737 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:51.908116 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:51.948694 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c17796e5d04014aafdac46984f4ba9 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:51.950721 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:52.005679 21793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:52.006322 29170 tablet_replica.cc:330] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:52.006614 29170 raft_consensus.cc:2229] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:52.006990 29170 raft_consensus.cc:2258] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:52.008076 29170 ts_tablet_manager.cc:1875] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:52.036725 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:52.068883 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.133306 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.137679 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:52.145805 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.157941 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:52.180548 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.201860 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:52.205762 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:52.217222 29170 ts_tablet_manager.cc:1888] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:20:52.217473 29170 log.cc:1196] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/12c17796e5d04014aafdac46984f4ba9
I20240502 08:20:52.218439 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 12c17796e5d04014aafdac46984f4ba9 (table rebalance_test_table_9 [id=6d0ce58fc888404ab140e25ec9c92a9b]) successfully deleted
I20240502 08:20:52.302073 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.304704 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:52.690768 21682 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:52.713697 23735 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:52.714738 28828 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:52.727092 21943 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:52.728072 28828 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:52.731957 29209 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:52.732625 21943 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:52.739998 21406 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:52.776855 23737 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:52.804808 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d933eacd944d84b39f0cb9c45641f8 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:52.821503 29306 tablet_replica.cc:330] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:52.821823 29306 raft_consensus.cc:2229] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:52.822069 29306 raft_consensus.cc:2258] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:52.822865 29306 ts_tablet_manager.cc:1875] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:52.851090 29306 ts_tablet_manager.cc:1888] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:52.853672 29306 log.cc:1196] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:20:52.855036 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet d4d933eacd944d84b39f0cb9c45641f8 (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
I20240502 08:20:52.952231 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.028789 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:53.038676 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.044814 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:53.079497 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:53.082044 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.108000 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.118480 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.252535 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.256278 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.262617 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:53.268896 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:53.300587 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:53.329104 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.354226 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.355273 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.463284 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.491503 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.495821 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:53.521230 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:53.522607 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:53.581589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:53.582099 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.619802 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:53.898999 21812 consensus_queue.cc:237] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:53.941496 23602 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:53.942530 29433 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:53.942744 21549 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:53.948531 21551 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:53.949398 29433 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:53.968626 23737 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:53.969405 28797 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:53.975332 23601 raft_consensus.cc:1261] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:53.976501 28797 consensus_queue.cc:1035] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:53.978996 29432 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:53.979565 23603 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:53.985081 21551 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:54.010320 21405 catalog_manager.cc:5484] T 296ccf9be99545789bbb3ff4031d3170 P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 4 to 5, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:54.013628 29522 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:54.046720 23737 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:54.052780 23601 raft_consensus.cc:2932] T e0241228bd204ca2a8ad2c57b6eeb543 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:54.081148 21402 catalog_manager.cc:5484] T e0241228bd204ca2a8ad2c57b6eeb543 P fd5e97ae7e344e4687b15ca0cde8d0ba reported cstate change: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:54.120900 21551 consensus_queue.cc:237] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:54.122381 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 296ccf9be99545789bbb3ff4031d3170 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:54.136566 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet e0241228bd204ca2a8ad2c57b6eeb543 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:54.144380 29546 tablet_replica.cc:330] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:54.144698 29546 raft_consensus.cc:2229] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:54.145443 29547 tablet_replica.cc:330] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:54.145700 29547 raft_consensus.cc:2229] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:54.145934 29547 raft_consensus.cc:2258] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:54.146867 29547 ts_tablet_manager.cc:1875] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:54.153949 29546 raft_consensus.cc:2258] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:54.154852 29546 ts_tablet_manager.cc:1875] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:54.159888 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:54.163763 29547 ts_tablet_manager.cc:1888] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:54.169700 29546 ts_tablet_manager.cc:1888] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:54.170022 29547 log.cc:1196] T e0241228bd204ca2a8ad2c57b6eeb543 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0241228bd204ca2a8ad2c57b6eeb543
I20240502 08:20:54.170369 29546 log.cc:1196] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/296ccf9be99545789bbb3ff4031d3170
I20240502 08:20:54.171455 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet e0241228bd204ca2a8ad2c57b6eeb543 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
I20240502 08:20:54.172150 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 296ccf9be99545789bbb3ff4031d3170 (table rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]) successfully deleted
I20240502 08:20:54.183467 23871 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:54.184080 21943 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:54.187206 29522 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:54.198478 29522 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:54.211792 23871 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:54.210248 29522 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:54.234175 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:54.251331 21406 catalog_manager.cc:5484] T d62f6eec9ee04f93a3738a83f790292c P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:54.259457 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet d62f6eec9ee04f93a3738a83f790292c with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:54.266664 21943 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:54.320830 29572 tablet_replica.cc:330] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:54.321393 29572 raft_consensus.cc:2229] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:54.321719 29572 raft_consensus.cc:2258] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:54.335981 29572 ts_tablet_manager.cc:1875] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:54.340299 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:54.381750 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:54.397818 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:54.406898 29572 ts_tablet_manager.cc:1888] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:54.415473 29572 log.cc:1196] T d62f6eec9ee04f93a3738a83f790292c P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/d62f6eec9ee04f93a3738a83f790292c
I20240502 08:20:54.416718 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet d62f6eec9ee04f93a3738a83f790292c (table rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]) successfully deleted
I20240502 08:20:54.501778 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:54.507992 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:54.554633 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:20:54.759744 21880 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:36745, user_credentials={real_user=slave}} blocked reactor thread for 48720.4us
I20240502 08:20:54.978439 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.024932 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:55.123494 21812 consensus_queue.cc:237] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:55.129900 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:55.130153 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.140950 23871 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:55.142159 28752 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:55.143527 21551 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:55.167915 29719 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:20:55.191581 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 49525.2us
I20240502 08:20:55.209954 28752 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:55.217057 23871 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:55.225773 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.230458 21402 catalog_manager.cc:5484] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:55.262951 21549 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:55.265301 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.266793 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:55.319442 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.333328 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.367992 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf79c69fd614284ba45fd052cac7ebf with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:55.384783 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:55.418885 29744 tablet_replica.cc:330] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:55.432068 29744 raft_consensus.cc:2229] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:55.432188 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:55.432492 29744 raft_consensus.cc:2258] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:55.434795 29744 ts_tablet_manager.cc:1875] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:55.453502 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.455588 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.480814 29744 ts_tablet_manager.cc:1888] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:55.480986 29744 log.cc:1196] T ebf79c69fd614284ba45fd052cac7ebf P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/ebf79c69fd614284ba45fd052cac7ebf
I20240502 08:20:55.481843 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet ebf79c69fd614284ba45fd052cac7ebf (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
I20240502 08:20:55.505481 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:55.522287 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.561049 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.668999 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:55.674466 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.697896 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:55.703888 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:55.704471 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.737265 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:55.770644 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.795001 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:55.956426 21943 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:55.976413 23871 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:55.977476 29149 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:55.991042 21812 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:55.992185 29149 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:56.010162 29149 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:56.014432 21811 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:56.030478 21403 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:56.049296 23870 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:56.095113 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet cec56facd7f740aeb42bc7c32495dae3 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49734
I20240502 08:20:56.141886 29867 tablet_replica.cc:330] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:20:56.142237 29867 raft_consensus.cc:2229] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:56.142716 29867 raft_consensus.cc:2258] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:56.144093 29867 ts_tablet_manager.cc:1875] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:56.152984 21943 consensus_queue.cc:237] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:56.169363 29867 ts_tablet_manager.cc:1888] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:56.170704 29867 log.cc:1196] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/cec56facd7f740aeb42bc7c32495dae3
I20240502 08:20:56.173904 23471 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:56.174916 29149 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:56.176247 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet cec56facd7f740aeb42bc7c32495dae3 (table rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]) successfully deleted
I20240502 08:20:56.183266 21682 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:56.184266 21812 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:20:56.185333 29633 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:56.186223 29633 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:20:56.239522 29149 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:56.244915 21811 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:56.251889 23471 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:56.252355 21406 catalog_manager.cc:5484] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:56.297683 21682 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:56.300752 23717 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b3ba1d834a4314b63f336675845441 with delete_type TABLET_DATA_TOMBSTONED (TS fd5e97ae7e344e4687b15ca0cde8d0ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33206
I20240502 08:20:56.312273 29910 tablet_replica.cc:330] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: stopping tablet replica
I20240502 08:20:56.312556 29910 raft_consensus.cc:2229] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:56.312732 29910 raft_consensus.cc:2258] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:56.313611 29910 ts_tablet_manager.cc:1875] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:56.330137 29910 ts_tablet_manager.cc:1888] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:20:56.335968 29910 log.cc:1196] T 80b3ba1d834a4314b63f336675845441 P fd5e97ae7e344e4687b15ca0cde8d0ba: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-6/wal/wals/80b3ba1d834a4314b63f336675845441
I20240502 08:20:56.337304 21397 catalog_manager.cc:4830] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): tablet 80b3ba1d834a4314b63f336675845441 (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:20:56.501892 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:56.530501 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:56.582775 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:56.595654 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:56.598332 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:56.714968 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:56.757215 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:56.800565 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:56.915539 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:56.926602 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:56.942385 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:56.991421 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.013628 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.051692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:57.090554 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:57.132467 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:57.176491 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.180837 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:57.328999 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:57.380499 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:57.437036 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.439520 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:57.450606 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:57.457269 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.474996 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.480238 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.487392 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:57.525063 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:57.542217 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.554778 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:57.573961 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:57.615779 21551 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:57.647878 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:57.652961 23603 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:20:57.653752 29700 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:20:57.725080 23603 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:57.736485 21402 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:57.722172 30103 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:20:57.793993 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 47926.3us
I20240502 08:20:57.844848 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0a886b3ba04737a6988c3b58c9698e with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:57.847518 21551 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
W20240502 08:20:57.857527 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c0a886b3ba04737a6988c3b58c9698e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:57.865080 23603 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:20:57.866427 29700 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:20:57.878954 30121 tablet_replica.cc:330] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:57.879302 30121 raft_consensus.cc:2229] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:57.881086 30121 raft_consensus.cc:2258] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:57.882277 30121 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:57.881835 29700 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:57.886433 23603 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:57.895447 30121 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:20:57.895828 30121 log.cc:1196] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:20:57.897532 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 9c0a886b3ba04737a6988c3b58c9698e (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
I20240502 08:20:57.900911 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9c0a886b3ba04737a6988c3b58c9698e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:20:57.908057 21402 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
W20240502 08:20:57.964980 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 46346.1us
I20240502 08:20:58.091480 23469 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:58.111088 21551 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:20:58.112463 28925 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:58.154753 21551 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:58.153658 30190 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:58.168998 21406 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:20:58.180063 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e7914da3f4494b239c6603513caee with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:58.180528 30121 tablet_replica.cc:330] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:58.180758 30121 raft_consensus.cc:2229] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:58.180997 30121 raft_consensus.cc:2258] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:58.181774 30121 ts_tablet_manager.cc:1875] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:58.190840 30121 ts_tablet_manager.cc:1888] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:20:58.191040 30121 log.cc:1196] T 203e7914da3f4494b239c6603513caee P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/203e7914da3f4494b239c6603513caee
I20240502 08:20:58.194940 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet 203e7914da3f4494b239c6603513caee (table rebalance_test_table_8 [id=0c5d895e104248b0a1ccfad7d1d590de]) successfully deleted
I20240502 08:20:58.207077 21682 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:58.229112 21683 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:20:58.230779 23737 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:58.231359 23870 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:20:58.231472 29986 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:58.233168 29986 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:20:58.250226 23603 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:58.250273 21551 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:20:58.251268 28828 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:58.252207 28828 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:20:58.260398 23469 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:58.264966 21549 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:20:58.266224 30197 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:20:58.270123 30197 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:58.276481 21549 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:58.300232 21406 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:20:58.305737 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 203e7914da3f4494b239c6603513caee with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:20:58.328802 30223 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:58.329854 23870 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:58.339819 30225 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:58.340965 30228 ts_tablet_manager.cc:902] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:20:58.346383 21551 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:58.355087 21406 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:58.362267 23603 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:20:58.362762 21402 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
W20240502 08:20:58.367976 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 99208.7us
I20240502 08:20:58.369391 30228 tablet_copy_client.cc:308] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:20:58.370121 23737 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:20:58.371628 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:57933)
I20240502 08:20:58.371790 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:57933: session id = 95f32202b9db457fb09872f6f51ea8ba-9c0a886b3ba04737a6988c3b58c9698e
W20240502 08:20:58.351742 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 203e7914da3f4494b239c6603513caee. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:20:58.375003 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57628
I20240502 08:20:58.375412 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet a20b2fd9f1ce488ba7793ed7ac268c16 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57062
I20240502 08:20:58.375530 21572 tablet_copy_source_session.cc:205] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:58.376027 30121 tablet_replica.cc:330] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:20:58.376258 30121 raft_consensus.cc:2229] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:58.376502 30121 raft_consensus.cc:2258] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:58.377269 30121 ts_tablet_manager.cc:1875] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:58.387418 30121 ts_tablet_manager.cc:1888] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:20:58.397217 30121 log.cc:1196] T a20b2fd9f1ce488ba7793ed7ac268c16 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/a20b2fd9f1ce488ba7793ed7ac268c16
I20240502 08:20:58.398408 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet a20b2fd9f1ce488ba7793ed7ac268c16 (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:20:58.401849 30238 tablet_replica.cc:330] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:20:58.402086 30238 raft_consensus.cc:2229] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:20:58.403334 30238 raft_consensus.cc:2258] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:20:58.405807 30238 ts_tablet_manager.cc:1875] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:20:58.413187 30238 ts_tablet_manager.cc:1888] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:20:58.413430 30238 log.cc:1196] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/d96a7174eb5442f2b0cc0cafc3cc4c80
I20240502 08:20:58.414371 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet d96a7174eb5442f2b0cc0cafc3cc4c80 (table rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]) successfully deleted
I20240502 08:20:58.427032 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:58.474884 30228 tablet_copy_client.cc:783] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:58.475876 30228 tablet_copy_client.cc:647] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:58.478482 30228 tablet_copy_client.cc:523] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:58.511227 30228 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:58.743404 30228 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:58.743914 30228 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:58.744205 30228 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.253s user 0.051s sys 0.013s
I20240502 08:20:58.759912 30228 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:58.760207 30228 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:58.760484 30228 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:58.761219 30228 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:20:58.762638 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-9c0a886b3ba04737a6988c3b58c9698e received from {username='slave'} at 127.8.68.200:57933
I20240502 08:20:58.762811 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-9c0a886b3ba04737a6988c3b58c9698e on tablet 9c0a886b3ba04737a6988c3b58c9698e with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:58.810366 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:58.826259 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:58.844664 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:58.876616 30228 ts_tablet_manager.cc:902] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:20:58.894459 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:58.906580 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:58.911610 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:58.943387 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:58.952176 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:58.974323 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.006126 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:59.036448 30228 tablet_copy_client.cc:308] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.197:45373
W20240502 08:20:59.038074 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 63347.8us
I20240502 08:20:59.057188 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.074452 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:59.079823 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.137145 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.138526 23491 tablet_copy_service.cc:140] P 20b637fbec5444dd91b3af9cdda482f5: Received BeginTabletCopySession request for tablet 203e7914da3f4494b239c6603513caee from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:51653)
I20240502 08:20:59.138775 23491 tablet_copy_service.cc:161] P 20b637fbec5444dd91b3af9cdda482f5: Beginning new tablet copy session on tablet 203e7914da3f4494b239c6603513caee from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:51653: session id = 95f32202b9db457fb09872f6f51ea8ba-203e7914da3f4494b239c6603513caee
I20240502 08:20:59.142881 23491 tablet_copy_source_session.cc:205] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:20:59.145720 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e7914da3f4494b239c6603513caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:20:59.152488 30228 tablet_copy_client.cc:783] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:20:59.152868 30228 tablet_copy_client.cc:647] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:20:59.156319 30228 tablet_copy_client.cc:523] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:20:59.160267 30228 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:20:59.181941 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:59.187778 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.238736 23870 raft_consensus.cc:1203] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:20:59.392338 30228 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:20:59.393015 30228 tablet_bootstrap.cc:492] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:20:59.393489 30228 ts_tablet_manager.cc:1367] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.233s user 0.059s sys 0.004s
I20240502 08:20:59.394606 30228 raft_consensus.cc:348] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:59.395017 30228 raft_consensus.cc:729] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:20:59.395860 30228 consensus_queue.cc:260] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:59.396723 30228 ts_tablet_manager.cc:1398] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:20:59.397791 23491 tablet_copy_service.cc:342] P 20b637fbec5444dd91b3af9cdda482f5: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-203e7914da3f4494b239c6603513caee received from {username='slave'} at 127.8.68.200:51653
I20240502 08:20:59.397940 23491 tablet_copy_service.cc:433] P 20b637fbec5444dd91b3af9cdda482f5: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-203e7914da3f4494b239c6603513caee on tablet 203e7914da3f4494b239c6603513caee with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:59.475200 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:59.593449 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:59.673674 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.732878 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:59.793812 23870 raft_consensus.cc:1203] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:20:59.844889 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.862941 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.877014 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:59.894904 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:20:59.941684 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:20:59.961584 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:20:59.965631 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:20:59.978950 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:20:59.984483 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:00.012777 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:00.039695 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:00.064894 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:00.146785 21683 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:00.171051 23737 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:00.172909 30351 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:00.221546 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:00.236617 30233 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.240164 23735 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.254247 21402 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 5 to 6, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.263948 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:00.267613 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:00.274690 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:00.281951 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:00.354189 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:00.369817 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:00.376945 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d933eacd944d84b39f0cb9c45641f8 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57062
I20240502 08:21:00.377813 21683 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:00.380903 23735 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:21:00.381130 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d933eacd944d84b39f0cb9c45641f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:00.381598 30233 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:00.384220 30233 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:00.389065 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d4d933eacd944d84b39f0cb9c45641f8 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:21:00.389772 23735 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:00.397316 30476 tablet_replica.cc:330] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:21:00.397583 21403 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba reported cstate change: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:00.408727 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:00.409401 30476 raft_consensus.cc:2229] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:00.409747 30476 raft_consensus.cc:2258] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:00.410573 30476 ts_tablet_manager.cc:1875] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:00.448735 30476 ts_tablet_manager.cc:1888] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:21:00.448881 30476 log.cc:1196] T d4d933eacd944d84b39f0cb9c45641f8 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:21:00.449939 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet d4d933eacd944d84b39f0cb9c45641f8 (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
I20240502 08:21:00.628643 21943 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:21:00.643622 23870 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:00.644955 30486 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:00.665040 30486 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:00.670058 23871 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:00.677747 21402 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:00.683091 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet cec56facd7f740aeb42bc7c32495dae3 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57628
I20240502 08:21:00.684960 21943 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:21:00.687847 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec56facd7f740aeb42bc7c32495dae3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:00.692966 23869 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:00.693193 30556 tablet_replica.cc:330] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:21:00.693414 30556 raft_consensus.cc:2229] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:00.693646 30556 raft_consensus.cc:2258] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:00.694455 30556 ts_tablet_manager.cc:1875] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:00.710943 30486 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:00.731715 30556 ts_tablet_manager.cc:1888] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:21:00.731882 30556 log.cc:1196] T cec56facd7f740aeb42bc7c32495dae3 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/cec56facd7f740aeb42bc7c32495dae3
I20240502 08:21:00.733448 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet cec56facd7f740aeb42bc7c32495dae3 (table rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]) successfully deleted
I20240502 08:21:00.737826 29996 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.767592 21943 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:00.774888 23869 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.783226 23469 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:00.784014 30575 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:00.785616 23735 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:00.786660 30575 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:00.791872 30575 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:00.799046 23469 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:00.824144 21402 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:00.832357 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with delete_type TABLET_DATA_TOMBSTONED (TS 13e764b27ed449c09aee719cdb2597f8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57628
I20240502 08:21:00.832867 30556 tablet_replica.cc:330] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: stopping tablet replica
I20240502 08:21:00.833180 30556 raft_consensus.cc:2229] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:00.833624 30556 raft_consensus.cc:2258] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:00.834694 30556 ts_tablet_manager.cc:1875] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:00.850775 30582 ts_tablet_manager.cc:902] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:00.852047 21402 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:00.893186 30556 ts_tablet_manager.cc:1888] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:21:00.893545 30556 log.cc:1196] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 13e764b27ed449c09aee719cdb2597f8: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:21:00.900759 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cec56facd7f740aeb42bc7c32495dae3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:21:00.910624 30582 tablet_copy_client.cc:308] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:00.913195 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:21:00.913375 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:21:00.904827 23735 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:00.916471 21703 tablet_copy_source_session.cc:205] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:00.920351 30582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:00.922240 21397 catalog_manager.cc:4830] TS 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
I20240502 08:21:00.954056 30582 tablet_copy_client.cc:783] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:00.954402 30582 tablet_copy_client.cc:647] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:00.967618 30582 tablet_copy_client.cc:523] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:00.972590 30582 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:21:00.997409 21683 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:01.001331 23469 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:01.001314 23735 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:01.001464 23869 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:01.002353 30351 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:01.002990 30351 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:01.004346 30233 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:01.026002 30233 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:01.027657 23469 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:01.036083 21403 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:01.038843 23871 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:01.042721 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39750
I20240502 08:21:01.060315 23735 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:01.067786 30635 tablet_replica.cc:330] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:21:01.068248 30635 raft_consensus.cc:2229] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:01.068835 30635 raft_consensus.cc:2258] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:01.070473 30635 ts_tablet_manager.cc:1875] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:01.090866 30635 ts_tablet_manager.cc:1888] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:21:01.093032 30635 log.cc:1196] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:21:01.094360 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:21:01.116508 30582 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:01.116994 30582 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:21:01.117318 30582 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.145s user 0.056s sys 0.008s
I20240502 08:21:01.118120 30582 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:01.118366 30582 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:21:01.118675 30582 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:01.125803 30582 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.008s user 0.002s sys 0.007s
I20240502 08:21:01.131855 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-d4d933eacd944d84b39f0cb9c45641f8 received from {username='slave'} at 127.8.68.198:60411
I20240502 08:21:01.131981 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-d4d933eacd944d84b39f0cb9c45641f8 on tablet d4d933eacd944d84b39f0cb9c45641f8 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:21:01.206166 30672 ts_tablet_manager.cc:902] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:21:01.304000 30672 tablet_copy_client.cc:308] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:21:01.305094 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet cec56facd7f740aeb42bc7c32495dae3 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:21:01.305296 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet cec56facd7f740aeb42bc7c32495dae3 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-cec56facd7f740aeb42bc7c32495dae3
I20240502 08:21:01.309464 21964 tablet_copy_source_session.cc:205] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:01.311129 30672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:01.317978 30672 tablet_copy_client.cc:783] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:01.318342 30672 tablet_copy_client.cc:647] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:01.320621 30672 tablet_copy_client.cc:523] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:01.326947 30672 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:21:01.403817 23601 raft_consensus.cc:1203] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:21:01.437841 30672 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:01.438809 30672 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:21:01.439355 30672 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.112s user 0.048s sys 0.009s
I20240502 08:21:01.440647 30672 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:01.447096 30672 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:21:01.447707 30672 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:01.455477 30672 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20240502 08:21:01.485380 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-cec56facd7f740aeb42bc7c32495dae3 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:21:01.485594 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-cec56facd7f740aeb42bc7c32495dae3 on tablet cec56facd7f740aeb42bc7c32495dae3 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:21:01.486335 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 46197.4us
I20240502 08:21:01.734910 23735 raft_consensus.cc:1203] T cec56facd7f740aeb42bc7c32495dae3 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:21:01.904920 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:01.925315 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:01.944654 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:01.951968 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:01.952663 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:01.962610 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:01.964459 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:01.990904 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.048884 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:02.069731 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:02.082065 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:02.097200 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.100848 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.119038 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:02.262835 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.335094 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.532155 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:02.542570 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.548763 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:02.573483 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:02.587726 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.616218 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:02.639827 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:02.650374 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:02.901571 21943 consensus_queue.cc:237] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:02.913420 23469 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:02.918126 30574 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:02.926573 21812 raft_consensus.cc:1261] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:02.929481 30574 consensus_queue.cc:1035] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:02.941002 30955 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:02.941948 23469 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:02.950562 21406 catalog_manager.cc:5484] T 80b3ba1d834a4314b63f336675845441 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:02.957954 21812 raft_consensus.cc:2932] T 80b3ba1d834a4314b63f336675845441 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:02.975121 21812 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:03.002372 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b3ba1d834a4314b63f336675845441 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49734
W20240502 08:21:03.004196 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc49c050280346db88d1323e30513c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:03.005753 23871 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:03.006613 30590 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:03.014416 23735 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:03.015318 30590 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:03.031386 31017 tablet_replica.cc:330] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:21:03.033625 31017 raft_consensus.cc:2229] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:03.035225 31017 raft_consensus.cc:2258] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:03.039572 31017 ts_tablet_manager.cc:1875] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:03.039327 30590 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:03.045893 23871 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:03.049448 21403 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:03.050524 23735 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:03.158053 31017 ts_tablet_manager.cc:1888] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:21:03.161429 31017 log.cc:1196] T 80b3ba1d834a4314b63f336675845441 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/80b3ba1d834a4314b63f336675845441
I20240502 08:21:03.163388 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 80b3ba1d834a4314b63f336675845441 (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
I20240502 08:21:03.261464 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:03.275523 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:03.290762 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:03.291491 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:03.292547 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:03.307767 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:03.359608 21943 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:03.360569 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:03.384699 23469 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:03.385571 30955 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:03.408077 30574 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:03.412292 23471 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:03.419668 21402 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:03.425101 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49734
I20240502 08:21:03.425664 31017 tablet_replica.cc:330] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:21:03.425861 31017 raft_consensus.cc:2229] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:03.426088 31017 raft_consensus.cc:2258] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:03.426910 31017 ts_tablet_manager.cc:1875] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:03.445974 31017 ts_tablet_manager.cc:1888] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:21:03.446210 31017 log.cc:1196] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/76bb1e3e2b9d4e31bf93ac1ce08a7833
I20240502 08:21:03.447172 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:21:03.443006 21943 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:21:03.451267 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76bb1e3e2b9d4e31bf93ac1ce08a7833. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:03.451598 23471 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:03.452237 30574 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:03.455324 30574 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:03.464995 23471 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:03.472517 21406 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:03.477463 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:21:03.483505 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:03.485795 31103 ts_tablet_manager.cc:902] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:21:03.567528 21683 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
W20240502 08:21:03.578104 21618 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Couldn't send request to peer 13e764b27ed449c09aee719cdb2597f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:03.585057 23601 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:03.585996 30233 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:03.593466 21943 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:03.594230 30351 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:03.599612 31103 tablet_copy_client.cc:308] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:21:03.601622 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet dc49c050280346db88d1323e30513c80 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:47771)
I20240502 08:21:03.601816 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet dc49c050280346db88d1323e30513c80 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:47771: session id = 4b9bfd2541144dad86507451343640d3-dc49c050280346db88d1323e30513c80
I20240502 08:21:03.604200 30351 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:21:03.606036 21833 tablet_copy_source_session.cc:205] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:03.607702 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc49c050280346db88d1323e30513c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:03.609176 23603 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:21:03.616700 21943 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:21:03.620630 31103 tablet_copy_client.cc:783] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:03.622085 31103 tablet_copy_client.cc:647] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:03.624523 21406 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:21:03.627133 31103 tablet_copy_client.cc:523] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:03.632254 31103 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:21:03.806334 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:03.824882 31103 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:03.825886 31103 tablet_bootstrap.cc:492] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:21:03.826334 31103 ts_tablet_manager.cc:1367] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.194s user 0.085s sys 0.030s
I20240502 08:21:03.829314 31103 raft_consensus.cc:348] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:03.831014 31103 raft_consensus.cc:729] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:21:03.831815 31103 consensus_queue.cc:260] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:03.840368 31103 ts_tablet_manager.cc:1398] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 08:21:03.842298 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-dc49c050280346db88d1323e30513c80 received from {username='slave'} at 127.8.68.198:47771
I20240502 08:21:03.842455 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 4b9bfd2541144dad86507451343640d3-dc49c050280346db88d1323e30513c80 on tablet dc49c050280346db88d1323e30513c80 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:21:03.851922 23603 raft_consensus.cc:1203] T dc49c050280346db88d1323e30513c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:21:03.887766 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:03.894641 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:03.916630 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:03.931563 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:03.958361 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:03.961908 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:03.995646 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:04.028932 31204 ts_tablet_manager.cc:902] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:21:04.045974 31213 ts_tablet_manager.cc:902] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:04.104490 31204 tablet_copy_client.cc:308] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:21:04.105742 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer fd5e97ae7e344e4687b15ca0cde8d0ba ({username='slave'} at 127.8.68.199:34395)
I20240502 08:21:04.105909 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer fd5e97ae7e344e4687b15ca0cde8d0ba at {username='slave'} at 127.8.68.199:34395: session id = fd5e97ae7e344e4687b15ca0cde8d0ba-76bb1e3e2b9d4e31bf93ac1ce08a7833
I20240502 08:21:04.109077 21964 tablet_copy_source_session.cc:205] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:04.112454 31204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:04.150009 31204 tablet_copy_client.cc:783] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:04.150765 31204 tablet_copy_client.cc:647] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:04.161630 31204 tablet_copy_client.cc:523] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:04.168980 31204 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap starting.
I20240502 08:21:04.175923 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:04.179064 21812 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:04.182020 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:04.189231 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:04.197773 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:04.229866 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:04.238216 23603 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:04.242547 30590 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:04.269805 21683 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:04.281200 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:04.290524 30590 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.297838 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:04.311354 31213 tablet_copy_client.cc:308] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:04.311651 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:04.313279 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet b88c4a1a36424086aad89b7585c17960 from peer 13e764b27ed449c09aee719cdb2597f8 ({username='slave'} at 127.8.68.195:60043)
I20240502 08:21:04.313473 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet b88c4a1a36424086aad89b7585c17960 from peer 13e764b27ed449c09aee719cdb2597f8 at {username='slave'} at 127.8.68.195:60043: session id = 13e764b27ed449c09aee719cdb2597f8-b88c4a1a36424086aad89b7585c17960
I20240502 08:21:04.318120 23603 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.319393 21703 tablet_copy_source_session.cc:205] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:04.324956 31213 ts_tablet_manager.cc:1875] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:21:04.331477 21406 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 reported cstate change: config changed from index 5 to 6, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:21:04.353279 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d96a7174eb5442f2b0cc0cafc3cc4c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:04.355453 21549 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:04.354825 23603 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:04.356205 30233 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:04.357363 31211 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:04.404556 30233 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:04.410296 21549 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:04.416522 23601 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:04.426513 31213 ts_tablet_manager.cc:1888] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:21:04.427206 31213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:04.436231 21812 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:04.438901 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 852613f14b324b38b5eecc29d4c602cd with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39750
W20240502 08:21:04.441224 21749 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:04.445124 23603 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:04.446260 31270 tablet_replica.cc:330] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:21:04.446337 31254 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:04.447459 31270 raft_consensus.cc:2229] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:04.454481 31270 raft_consensus.cc:2258] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:04.458070 31270 ts_tablet_manager.cc:1875] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:04.459012 31254 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:04.459952 23603 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:04.464640 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 852613f14b324b38b5eecc29d4c602cd with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:21:04.475744 21403 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:04.476325 21407 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:04.526501 31270 ts_tablet_manager.cc:1888] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:21:04.526744 31270 log.cc:1196] T 852613f14b324b38b5eecc29d4c602cd P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/852613f14b324b38b5eecc29d4c602cd
I20240502 08:21:04.533419 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet 852613f14b324b38b5eecc29d4c602cd (table rebalance_test_table_6 [id=fe5182679a84496691f023e5b676fd50]) successfully deleted
I20240502 08:21:04.535760 31213 tablet_copy_client.cc:783] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:04.536206 31213 tablet_copy_client.cc:647] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:04.540844 31213 tablet_copy_client.cc:523] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:04.567579 31213 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:21:04.633056 31204 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:04.633543 31204 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Bootstrap complete.
I20240502 08:21:04.633834 31204 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent bootstrapping tablet: real 0.465s user 0.097s sys 0.021s
I20240502 08:21:04.635846 31204 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:04.636087 31204 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5e97ae7e344e4687b15ca0cde8d0ba, State: Initialized, Role: FOLLOWER
I20240502 08:21:04.637568 31204 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:04.645727 31204 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20240502 08:21:04.652410 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-76bb1e3e2b9d4e31bf93ac1ce08a7833 received from {username='slave'} at 127.8.68.199:34395
I20240502 08:21:04.652563 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session fd5e97ae7e344e4687b15ca0cde8d0ba-76bb1e3e2b9d4e31bf93ac1ce08a7833 on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with peer fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:21:04.766382 31213 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:04.786716 21683 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:04.812153 23869 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:21:04.812987 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20b2fd9f1ce488ba7793ed7ac268c16. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:04.814446 30233 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:04.815925 23735 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:04.816572 31213 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap complete.
I20240502 08:21:04.818167 30233 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:04.818774 31213 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.251s user 0.077s sys 0.006s
I20240502 08:21:04.819952 31213 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:21:04.821784 31213 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:21:04.822319 31213 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:21:04.824481 31213 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:21:04.827682 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 13e764b27ed449c09aee719cdb2597f8-b88c4a1a36424086aad89b7585c17960 received from {username='slave'} at 127.8.68.195:60043
I20240502 08:21:04.827844 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 13e764b27ed449c09aee719cdb2597f8-b88c4a1a36424086aad89b7585c17960 on tablet b88c4a1a36424086aad89b7585c17960 with peer 13e764b27ed449c09aee719cdb2597f8
I20240502 08:21:04.885699 30233 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.890792 23871 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.904706 23735 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.907307 21403 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:04.934396 31358 ts_tablet_manager.cc:902] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:04.967085 23735 raft_consensus.cc:1203] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:21:05.000881 31358 tablet_copy_client.cc:308] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:05.001838 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:21:05.002058 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-d96a7174eb5442f2b0cc0cafc3cc4c80
I20240502 08:21:05.007097 21703 tablet_copy_source_session.cc:205] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:05.010483 31358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:05.019754 31358 tablet_copy_client.cc:783] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:05.020074 31358 tablet_copy_client.cc:647] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:05.031864 31358 tablet_copy_client.cc:523] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:05.035849 31358 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:21:05.064411 31400 ts_tablet_manager.cc:902] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:21:05.099874 31400 tablet_copy_client.cc:308] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:21:05.101063 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 852613f14b324b38b5eecc29d4c602cd from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:57225)
I20240502 08:21:05.101254 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 852613f14b324b38b5eecc29d4c602cd from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:57225: session id = 0f655ab8f20d4ba482c3173f9594b98b-852613f14b324b38b5eecc29d4c602cd
I20240502 08:21:05.106271 21833 tablet_copy_source_session.cc:205] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:05.109413 21812 raft_consensus.cc:1203] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:21:05.113374 31400 ts_tablet_manager.cc:1875] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:21:05.167181 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:05.178117 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:05.233363 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:05.235150 31400 ts_tablet_manager.cc:1888] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:21:05.235505 31400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852613f14b324b38b5eecc29d4c602cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:05.251307 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:05.263506 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:05.263837 31400 tablet_copy_client.cc:783] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:05.264124 31400 tablet_copy_client.cc:647] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:05.266054 31400 tablet_copy_client.cc:523] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:05.270126 31400 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:21:05.281054 31444 ts_tablet_manager.cc:902] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:05.371623 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:05.377130 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:05.421924 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:05.424566 31444 tablet_copy_client.cc:308] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:05.427685 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer 4b9bfd2541144dad86507451343640d3 ({username='slave'} at 127.8.68.198:60411)
I20240502 08:21:05.427930 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet a20b2fd9f1ce488ba7793ed7ac268c16 from peer 4b9bfd2541144dad86507451343640d3 at {username='slave'} at 127.8.68.198:60411: session id = 4b9bfd2541144dad86507451343640d3-a20b2fd9f1ce488ba7793ed7ac268c16
I20240502 08:21:05.432107 21703 tablet_copy_source_session.cc:205] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:05.434028 31444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20b2fd9f1ce488ba7793ed7ac268c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:05.459146 31358 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:05.461161 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:05.469370 31444 tablet_copy_client.cc:783] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:05.469722 31444 tablet_copy_client.cc:647] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:05.473630 31444 tablet_copy_client.cc:523] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:05.483681 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:05.489954 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:05.491055 31358 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:21:05.491473 31358 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.456s user 0.070s sys 0.030s
I20240502 08:21:05.492877 31444 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Bootstrap starting.
I20240502 08:21:05.497364 31358 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:05.501013 31358 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:21:05.505137 31358 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:05.516551 31358 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240502 08:21:05.529795 21662 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:21:05.533288 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d96a7174eb5442f2b0cc0cafc3cc4c80 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:21:05.533479 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d96a7174eb5442f2b0cc0cafc3cc4c80 on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:21:05.551789 23851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:05.554656 21812 consensus_queue.cc:237] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:05.571396 23717 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:21:05.596426 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf79c69fd614284ba45fd052cac7ebf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:05.597635 23871 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:05.599321 21549 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:05.599792 31455 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:05.600829 31455 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:05.613286 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:05.624641 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:05.645391 31455 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:05.648437 23871 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:05.656162 21549 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:05.659288 21406 catalog_manager.cc:5484] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:05.817906 31400 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:05.817898 23471 raft_consensus.cc:1203] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:21:05.818368 31400 tablet_bootstrap.cc:492] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:21:05.818670 31400 ts_tablet_manager.cc:1367] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.549s user 0.054s sys 0.004s
I20240502 08:21:05.843863 31444 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:05.844535 31444 tablet_bootstrap.cc:492] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Bootstrap complete.
I20240502 08:21:05.844938 31444 ts_tablet_manager.cc:1367] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Time spent bootstrapping tablet: real 0.352s user 0.103s sys 0.094s
I20240502 08:21:05.846074 31444 raft_consensus.cc:348] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:05.846554 31444 raft_consensus.cc:729] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bfd2541144dad86507451343640d3, State: Initialized, Role: FOLLOWER
I20240502 08:21:05.846904 31444 consensus_queue.cc:260] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:05.855219 31444 ts_tablet_manager.cc:1398] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20240502 08:21:05.859859 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 4b9bfd2541144dad86507451343640d3-a20b2fd9f1ce488ba7793ed7ac268c16 received from {username='slave'} at 127.8.68.198:60411
I20240502 08:21:05.860013 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 4b9bfd2541144dad86507451343640d3-a20b2fd9f1ce488ba7793ed7ac268c16 on tablet a20b2fd9f1ce488ba7793ed7ac268c16 with peer 4b9bfd2541144dad86507451343640d3
I20240502 08:21:05.866720 31400 raft_consensus.cc:348] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:05.867172 31400 raft_consensus.cc:729] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:21:05.867518 31400 consensus_queue.cc:260] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:05.868264 31400 ts_tablet_manager.cc:1398] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20240502 08:21:05.869051 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-852613f14b324b38b5eecc29d4c602cd received from {username='slave'} at 127.8.68.196:57225
I20240502 08:21:05.869191 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-852613f14b324b38b5eecc29d4c602cd on tablet 852613f14b324b38b5eecc29d4c602cd with peer 0f655ab8f20d4ba482c3173f9594b98b
I20240502 08:21:05.880084 21549 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
W20240502 08:21:05.891631 21488 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:05.892376 23871 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:21:05.896237 31491 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:21:05.904394 23601 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:21:05.905134 31491 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:21:05.946981 31527 log.cc:924] Time spent T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240502 08:21:05.951718 31491 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:05.956190 23869 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:05.966365 21403 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 9 to 10, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:05.970912 23601 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:21:06.087662 21944 raft_consensus.cc:1203] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:21:06.089360 21683 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa"
dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44226
I20240502 08:21:06.089592 21683 raft_consensus.cc:595] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:21:06.096465 31577 ts_tablet_manager.cc:902] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:21:06.115509 31577 tablet_copy_client.cc:308] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:21:06.116536 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet ebf79c69fd614284ba45fd052cac7ebf from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:53979)
I20240502 08:21:06.116755 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet ebf79c69fd614284ba45fd052cac7ebf from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:53979: session id = 20b637fbec5444dd91b3af9cdda482f5-ebf79c69fd614284ba45fd052cac7ebf
I20240502 08:21:06.120704 21833 tablet_copy_source_session.cc:205] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:06.125283 31577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf79c69fd614284ba45fd052cac7ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:06.157339 31577 tablet_copy_client.cc:783] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:06.157626 31577 tablet_copy_client.cc:647] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:06.174453 31577 tablet_copy_client.cc:523] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:06.186667 31577 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
I20240502 08:21:06.204986 21812 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
W20240502 08:21:06.217031 21748 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bfc3eaed583458b8381d4e7b4d36aef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:06.218358 23601 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:06.218994 31455 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:06.226326 21549 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:21:06.227149 31455 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:21:06.263029 31255 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:06.263757 21551 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:06.270162 21403 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:06.274909 23601 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:06.289403 23603 raft_consensus.cc:1203] T a20b2fd9f1ce488ba7793ed7ac268c16 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:21:06.315127 31336 raft_consensus.cc:979] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: : Instructing follower 95f32202b9db457fb09872f6f51ea8ba to start an election
I20240502 08:21:06.315299 31336 raft_consensus.cc:1067] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Signalling peer 95f32202b9db457fb09872f6f51ea8ba to start an election
I20240502 08:21:06.317736 23869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa"
dest_uuid: "95f32202b9db457fb09872f6f51ea8ba"
from {username='slave'} at 127.8.68.194:44749
I20240502 08:21:06.318082 23869 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:21:06.318279 23869 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:21:06.348280 23869 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:06.351900 31400 ts_tablet_manager.cc:902] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
I20240502 08:21:06.358984 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "95f32202b9db457fb09872f6f51ea8ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:21:06.359354 21683 raft_consensus.cc:3032] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:21:06.359453 21683 raft_consensus.cc:729] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:21:06.359771 21683 consensus_queue.cc:260] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:06.360306 21683 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:21:06.363984 21683 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f32202b9db457fb09872f6f51ea8ba in term 2.
I20240502 08:21:06.365955 23471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "95f32202b9db457fb09872f6f51ea8ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "20b637fbec5444dd91b3af9cdda482f5"
I20240502 08:21:06.366272 23471 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:21:06.369127 23471 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f32202b9db457fb09872f6f51ea8ba in term 2.
I20240502 08:21:06.370263 23806 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5, 60677121510e47a5b51d1eb22f9aa238, 95f32202b9db457fb09872f6f51ea8ba; no voters:
I20240502 08:21:06.370783 31488 raft_consensus.cc:2781] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:21:06.382714 23869 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [CANDIDATE]: Term 2 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397)
I20240502 08:21:06.388073 31400 tablet_copy_client.cc:308] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:21:06.389200 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:52239)
I20240502 08:21:06.389430 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:52239: session id = 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:21:06.390027 31488 raft_consensus.cc:686] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 LEADER]: Becoming Leader. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Running, Role: LEADER
I20240502 08:21:06.390702 31488 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:06.392619 21572 tablet_copy_source_session.cc:205] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:06.394112 31400 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:21:06.402715 31400 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 08:21:06.403172 31400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:06.407228 21406 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: term changed from 1 to 2, leader changed from 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) to 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200). New cstate: current_term: 2 leader_uuid: "95f32202b9db457fb09872f6f51ea8ba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } }
I20240502 08:21:06.421038 23735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f04a3972e384fa3b0ffe0cb8a4f35aa" candidate_uuid: "95f32202b9db457fb09872f6f51ea8ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba"
I20240502 08:21:06.421303 23735 raft_consensus.cc:3037] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:21:06.425434 23735 raft_consensus.cc:2454] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f32202b9db457fb09872f6f51ea8ba in term 2.
I20240502 08:21:06.443221 31577 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:06.443679 31577 tablet_bootstrap.cc:492] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:21:06.443969 31577 ts_tablet_manager.cc:1367] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.257s user 0.058s sys 0.014s
I20240502 08:21:06.444701 31577 raft_consensus.cc:348] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:06.444931 31577 raft_consensus.cc:729] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:21:06.446825 31400 tablet_copy_client.cc:783] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:06.447079 31400 tablet_copy_client.cc:647] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:06.450435 31400 tablet_copy_client.cc:523] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:06.455089 31400 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:21:06.463548 31577 consensus_queue.cc:260] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:06.469275 31577 ts_tablet_manager.cc:1398] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 08:21:06.476358 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-ebf79c69fd614284ba45fd052cac7ebf received from {username='slave'} at 127.8.68.197:53979
I20240502 08:21:06.476552 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-ebf79c69fd614284ba45fd052cac7ebf on tablet ebf79c69fd614284ba45fd052cac7ebf with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:21:06.538759 23471 raft_consensus.cc:1203] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:21:06.606380 21683 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:21:06.610195 23735 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:06.611042 23603 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:21:06.611532 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182a6f2df91242cfa10f05ef788a19cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:06.612695 30233 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:06.615633 30233 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:06.658390 30233 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:06.660202 23735 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:06.667819 23601 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:06.673125 21407 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P fd5e97ae7e344e4687b15ca0cde8d0ba reported cstate change: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:21:06.682700 21683 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:21:06.696035 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d933eacd944d84b39f0cb9c45641f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:06.697165 23603 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:06.698004 23735 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:21:06.698282 31716 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:06.698649 30233 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:21:06.722291 30233 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:06.730190 31730 ts_tablet_manager.cc:902] T 2bfc3eaed583458b8381d4e7b4d36aef P fd5e97ae7e344e4687b15ca0cde8d0ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
F20240502 08:21:06.730635 31730 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.queue_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.queue_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_cpu_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_cpu_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_wall_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_wall_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.queue_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.queue_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_cpu_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_cpu_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_wall_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_wall_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us) (tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us, tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.queue_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.queue_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_cpu_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_cpu_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_wall_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_wall_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.queue_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.queue_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_cpu_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_cpu_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_wall_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_wall_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.queue_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.queue_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_cpu_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_cpu_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_wall_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_wall_time_us) (tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.queue_time_us, tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.queue_time_us) (tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.run_cpu_time_us, tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.run_cpu_time_us) (tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.run_wall_time_us, tablet-download-pool-7b3f6a0b2561419ca6bd832fd65e3196.run_wall_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.queue_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.queue_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_cpu_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_cpu_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_wall_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_wall_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.queue_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.queue_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_cpu_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_cpu_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_wall_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_wall_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.queue_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.queue_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_cpu_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_cpu_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_wall_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 08:21:06.735502 23735 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P fd5e97ae7e344e4687b15ca0cde8d0ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:06.737540 23603 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 4b9bfd2541144dad86507451343640d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:06.738123 21403 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 7 to 8, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
@ 0x7f967879fdcd google::LogMessage::Fail() at ??:0
@ 0x7f96787a3b93 google::LogMessage::SendToLog() at ??:0
I20240502 08:21:06.802568 31400 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:06.803067 31400 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:21:06.804167 31400 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.349s user 0.058s sys 0.007s
I20240502 08:21:06.806274 31400 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:06.806489 31400 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:21:06.807308 31400 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:06.808645 31400 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
@ 0x7f967879f7cc google::LogMessage::Flush() at ??:0
I20240502 08:21:06.809559 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e received from {username='slave'} at 127.8.68.196:52239
I20240502 08:21:06.809720 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-9c0a886b3ba04737a6988c3b58c9698e on tablet 9c0a886b3ba04737a6988c3b58c9698e with peer 0f655ab8f20d4ba482c3173f9594b98b
@ 0x7f96787a0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f967b4e0742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:21:06.842113 23736 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P fd5e97ae7e344e4687b15ca0cde8d0ba [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:21:06.843449 31488 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f967b4acf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:21:06.867405 21944 raft_consensus.cc:1203] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 08:21:06.877010 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7f967b4a6cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f968a8f98de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:21:06.898712 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:06.899623 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:06.907804 23471 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:21:06.908536 31488 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f968a911daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:21:06.923664 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
@ 0x7f968aa3c480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f968aa74a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f968aa57981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f968aa5770e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:21:06.963968 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:07.004072 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:07.020385 21683 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
@ 0x7f967b4b0b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f967b4b419c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f967b4b4019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f967b48f01e kudu::Thread::SuperviseThread() at ??:0
I20240502 08:21:07.064481 31488 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f967e8146db start_thread at ??:0
@ 0x7f967765271f clone at ??:0
I20240502 08:21:07.080987 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:07.090680 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:07.104715 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:07.128587 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:07.165786 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:07.172835 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:07.202409 23583 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:21:07.231688 31816 ts_tablet_manager.cc:902] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:07.232682 31817 ts_tablet_manager.cc:902] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:21:07.238412 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:07.312651 31816 tablet_copy_client.cc:308] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:07.313488 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 20b637fbec5444dd91b3af9cdda482f5 ({username='slave'} at 127.8.68.197:35657)
I20240502 08:21:07.313658 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 20b637fbec5444dd91b3af9cdda482f5 at {username='slave'} at 127.8.68.197:35657: session id = 20b637fbec5444dd91b3af9cdda482f5-d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:21:07.313726 23471 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:21:07.316746 21703 tablet_copy_source_session.cc:205] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:07.323799 31816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:07.329895 23869 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:21:07.330988 30197 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:21:07.332737 21551 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:21:07.333923 30197 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:21:07.339854 30197 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:07.344213 21551 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:07.349802 31817 tablet_copy_client.cc:308] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:21:07.351187 21408 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 8 to 9, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:07.367660 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 182a6f2df91242cfa10f05ef788a19cb from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:42037)
I20240502 08:21:07.367918 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 182a6f2df91242cfa10f05ef788a19cb from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:42037: session id = 95f32202b9db457fb09872f6f51ea8ba-182a6f2df91242cfa10f05ef788a19cb
I20240502 08:21:07.375864 23869 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:21:07.377295 21703 tablet_copy_source_session.cc:205] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:21:07.394847 31817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182a6f2df91242cfa10f05ef788a19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:21:07.400321 31816 tablet_copy_client.cc:783] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 0 data blocks...
W20240502 08:21:07.409217 23405 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 203e7914da3f4494b239c6603513caee. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:07.415573 31816 tablet_copy_client.cc:647] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:07.417793 31816 tablet_copy_client.cc:523] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:07.428198 31816 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap starting.
W20240502 08:21:07.441200 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 61010.8us
W20240502 08:21:07.443967 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 80590us
I20240502 08:21:07.500349 31817 tablet_copy_client.cc:783] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 0 data blocks...
I20240502 08:21:07.500664 31817 tablet_copy_client.cc:647] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:21:07.503914 31817 tablet_copy_client.cc:523] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:21:07.516845 31817 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:21:07.598443 31816 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:07.598883 31816 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Bootstrap complete.
I20240502 08:21:07.599136 31816 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Time spent bootstrapping tablet: real 0.171s user 0.056s sys 0.001s
I20240502 08:21:07.607447 31817 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:21:07.607946 31817 tablet_bootstrap.cc:492] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:21:07.608245 31817 ts_tablet_manager.cc:1367] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.093s user 0.059s sys 0.004s
I20240502 08:21:07.608428 31816 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:07.608646 31816 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b637fbec5444dd91b3af9cdda482f5, State: Initialized, Role: FOLLOWER
I20240502 08:21:07.608884 31816 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:07.609066 31817 raft_consensus.cc:348] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:21:07.609321 31817 raft_consensus.cc:729] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:21:07.609408 31816 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:21:07.610443 31817 consensus_queue.cc:260] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:21:07.611392 31817 ts_tablet_manager.cc:1398] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:21:07.612126 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-182a6f2df91242cfa10f05ef788a19cb received from {username='slave'} at 127.8.68.200:42037
I20240502 08:21:07.612253 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-182a6f2df91242cfa10f05ef788a19cb on tablet 182a6f2df91242cfa10f05ef788a19cb with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:21:07.618080 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d4d933eacd944d84b39f0cb9c45641f8 received from {username='slave'} at 127.8.68.197:35657
I20240502 08:21:07.618229 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 20b637fbec5444dd91b3af9cdda482f5-d4d933eacd944d84b39f0cb9c45641f8 on tablet d4d933eacd944d84b39f0cb9c45641f8 with peer 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:21:07.742239 23871 raft_consensus.cc:1203] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:21:07.791347 31904 ts_tablet_manager.cc:902] T 203e7914da3f4494b239c6603513caee P 4b9bfd2541144dad86507451343640d3: Initiating tablet copy from peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
F20240502 08:21:07.791697 31904 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us) (tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.queue_time_us, tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.queue_time_us) (tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.run_cpu_time_us, tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.run_cpu_time_us) (tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.run_wall_time_us, tablet-download-pool-296ccf9be99545789bbb3ff4031d3170.run_wall_time_us) (tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us, tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us) (tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.run_cpu_time_us, tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.run_cpu_time_us) (tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.run_wall_time_us, tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.run_wall_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us) (tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.queue_time_us, tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.queue_time_us) (tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.run_cpu_time_us, tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.run_cpu_time_us) (tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.run_wall_time_us, tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.run_wall_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.queue_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.queue_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_cpu_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_cpu_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_wall_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_wall_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.queue_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.queue_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_cpu_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_cpu_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_wall_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_wall_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us) (tablet-download-pool-b88c4a1a36424086aad89b7585c17960.queue_time_us, tablet-download-pool-b88c4a1a36424086aad89b7585c17960.queue_time_us) (tablet-download-pool-b88c4a1a36424086aad89b7585c17960.run_cpu_time_us, tablet-download-pool-b88c4a1a36424086aad89b7585c17960.run_cpu_time_us) (tablet-download-pool-b88c4a1a36424086aad89b7585c17960.run_wall_time_us, tablet-download-pool-b88c4a1a36424086aad89b7585c17960.run_wall_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.queue_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.queue_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_cpu_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_cpu_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_wall_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_wall_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.queue_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.queue_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_cpu_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_cpu_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_wall_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_wall_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.queue_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.queue_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_cpu_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_cpu_time_us) (tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_wall_time_us, tablet-download-pool-e0241228bd204ca2a8ad2c57b6eeb543.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9d59765dcd google::LogMessage::Fail() at ??:0
@ 0x7f9d59769b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9d597657cc google::LogMessage::Flush() at ??:0
@ 0x7f9d59766f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9d5c4a6742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f9d5c472f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f9d5c46ccd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f9d6b8bf8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f9d6b8d7daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f9d6ba02480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f9d6ba3aa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f9d6ba1d981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f9d6ba1d70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9d5c476b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9d5c47a19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9d5c47a019 std::_Function_handler<>::_M_invoke() at ??:0
W20240502 08:21:07.892765 23406 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.893111 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:21:07.897171 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:07.901611 23806 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.901901 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240502 08:21:07.907796 21487 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.908121 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f9d5c45501e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9d5f7da6db start_thread at ??:0
W20240502 08:21:07.918339 21879 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.918677 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f9d5861871f clone at ??:0
W20240502 08:21:07.928107 21618 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.928455 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:21:07.929062 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:07.929231 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:07.944139 21748 connection.cc:535] client connection to 127.8.68.199:33397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.944500 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:07.975524 21487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 67484.5us
W20240502 08:21:07.988801 31926 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:07.906513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.199:33397 (local address 127.8.68.200:52203)
0502 08:21:07.948378 (+ 41865us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:07.948403 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:07.948559 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:07.987532 (+ 38973us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41718,"thread_start_us":42150,"threads_started":1}
W20240502 08:21:07.989075 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:07.996088 31931 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:07.935795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.199:33397 (local address 127.8.68.197:37599)
0502 08:21:07.994846 (+ 59051us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:07.994983 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58954,"thread_start_us":58874,"threads_started":1}
W20240502 08:21:07.998380 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:07.999337 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 80760.1us
W20240502 08:21:08.002481 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.002635 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:08.003439 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.003571 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.005882 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 77511.4us
I20240502 08:21:08.007112 23471 raft_consensus.cc:1203] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:21:08.017760 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:08.041610 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:08.043977 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.044169 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.044261 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.063992 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.064154 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.064250 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.064334 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.068936 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:21:08.086733 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:08.202149 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:08.555791 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:08.560899 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:08.607142 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:08.612541 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:08.628250 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:08.643957 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:08.664624 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:08.676542 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:08.719821 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:08.736385 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:08.776032 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:08.783519 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:08.792310 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:08.822962 21747 connection.cc:535] client connection to 127.8.68.198:41299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:08.823629 21486 connection.cc:535] client connection to 127.8.68.198:41299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:08.824076 21617 connection.cc:535] client connection to 127.8.68.198:41299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:21:08.830540 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.830736 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.830821 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831045 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831175 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831359 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831478 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831598 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831687 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.831785 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.832020 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.832166 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.832265 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.832351 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:21:08.873416 23405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 51292.2us
W20240502 08:21:08.891173 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:09.074249 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:09.133185 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:09.155843 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:09.192492 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:09.201696 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:09.205443 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:09.217285 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:09.237704 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:09.323050 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:09.328497 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:09.356422 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:09.366641 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:09.392400 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:09.434376 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:09.617528 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:09.626224 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:09.919486 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:09.949054 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:09.954718 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:09.960628 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:09.989025 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:10.017097 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.043184 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:10.086905 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:10.103051 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:10.157169 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:10.160590 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:10.183702 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.185721 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.185843 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:10.189571 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:10.226495 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:10.230795 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 51647us
W20240502 08:21:10.234994 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:10.246022 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:10.260808 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:10.288933 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:10.309335 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:10.319918 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:10.323417 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:10.345921 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:10.388774 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:10.390736 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:10.394788 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:10.442847 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.500821 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.523178 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.548235 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:10.573272 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.652189 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.668753 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.673182 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.718706 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.792317 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:10.912897 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:10.937654 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:10.964278 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:11.032354 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:11.038630 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:11.083039 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:11.091356 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:11.100533 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:11.118901 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.132097 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.136761 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.190960 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:11.212934 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:11.233161 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:11.249802 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.252782 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.257432 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 66887.8us
I20240502 08:21:11.288923 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:11.312134 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.350117 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:21:11.529824 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:11.537803 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:21:11.580988 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:11.590691 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:11.635908 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:11.647135 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:11.665040 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:11.666446 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:11.784811 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:11.815996 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:11.831372 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:11.832635 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:11.840919 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:11.843493 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:11.867928 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:11.880640 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:11.907766 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:11.966125 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:12.002396 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:12.010167 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:12.028558 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:12.104893 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:12.162542 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:12.196480 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:12.213131 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:12.236548 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:12.250942 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:12.256362 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:12.313051 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:12.325443 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 121360us
W20240502 08:21:12.336788 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.435283 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.686167 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.717762 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.726162 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:12.734205 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.744383 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.777637 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.783875 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.881333 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.894719 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:12.903745 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:12.981568 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 08:21:12.983186 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.015102 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:13.028573 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:13.029624 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 08:21:13.051474 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:13.062187 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:21:13.069664 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.073401 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:21:13.074012 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:21:13.108361 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:13.112179 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:13.133037 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.143674 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:13.152720 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:13.153715 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.161403 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:13.193147 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:21:13.195171 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:13.204823 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.287006 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:13.325628 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:13.331588 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:13.348985 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:13.395501 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.401086 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:13.436945 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.457862 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:13.484233 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.510368 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.552340 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:13.569960 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:13.590720 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:13.594017 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:13.596406 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.601032 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:13.606444 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:13.613471 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:13.625311 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:13.646600 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:13.673969 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:13.694803 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:13.713289 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:13.790853 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.816198 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.817461 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:13.817610 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:13.839663 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:13.870229 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:13.870815 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:14.000420 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 121008us
W20240502 08:21:14.001524 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:14.002673 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:14.064447 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:14.076403 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:14.103125 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:21:14.112082 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:21:14.136274 21811 consensus_queue.cc:237] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:21:14.136646 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:14.164899 21549 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:14.164884 23470 raft_consensus.cc:1261] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:21:14.165566 383 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:21:14.165939 32697 consensus_queue.cc:1035] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:21:14.178823 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:14.189465 383 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:14.195614 21549 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:14.196483 23471 raft_consensus.cc:2932] T ebf79c69fd614284ba45fd052cac7ebf P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:14.202172 21407 catalog_manager.cc:5484] T ebf79c69fd614284ba45fd052cac7ebf P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:21:14.342396 23851 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf79c69fd614284ba45fd052cac7ebf with delete_type TABLET_DATA_TOMBSTONED (TS 95f32202b9db457fb09872f6f51ea8ba not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53854
I20240502 08:21:14.371644 456 tablet_replica.cc:330] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: stopping tablet replica
I20240502 08:21:14.372262 456 raft_consensus.cc:2229] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:21:14.372614 456 raft_consensus.cc:2258] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:21:14.374082 456 ts_tablet_manager.cc:1875] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:21:14.382036 456 ts_tablet_manager.cc:1888] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:21:14.382362 456 log.cc:1196] T ebf79c69fd614284ba45fd052cac7ebf P 95f32202b9db457fb09872f6f51ea8ba: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-7/wal/wals/ebf79c69fd614284ba45fd052cac7ebf
I20240502 08:21:14.383556 21395 catalog_manager.cc:4830] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): tablet ebf79c69fd614284ba45fd052cac7ebf (table rebalance_test_table_0 [id=6d3160598c3b46be94de175cb8b05c0a]) successfully deleted
W20240502 08:21:14.401831 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:14.632961 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:14.677829 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:14.713099 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:14.737197 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:14.748471 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:14.775682 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:14.785511 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:14.819166 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:14.836788 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:14.889008 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 126598us
I20240502 08:21:14.889978 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:14.973649 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:14.985314 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:14.994974 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.016497 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:15.032528 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:15.047777 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:15.076539 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.100854 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:15.131156 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:15.173584 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:15.183036 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.185676 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:15.199502 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:15.241708 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:15.252319 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:15.253636 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.260927 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:15.261878 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:15.267799 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.270195 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.288383 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:15.293077 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:15.297001 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:15.298094 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:15.310506 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:15.316066 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:15.349725 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.359704 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:15.484076 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.510419 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.639681 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:15.639686 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.717177 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.745942 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:15.769104 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.794515 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.911260 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:15.929643 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:15.941267 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:15.953863 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:15.965355 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:15.987962 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.000617 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:16.023129 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:16.027545 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.040812 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:16.133224 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:16.184432 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:16.201807 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.219625 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:16.223668 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:16.270615 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:16.290633 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:16.326937 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:21:16.328838 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.338730 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.359766 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:16.407931 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.522920 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.526898 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:16.573908 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.578348 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:16.581916 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.607409 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:16.609737 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:16.632493 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:21:16.634969 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:16.643532 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:16.646311 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.659049 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:16.672130 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:16.770069 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:16.771045 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.807992 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:16.815950 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:16.859156 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:16.870090 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.874903 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:16.878903 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:16.948557 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:16.953860 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:17.043972 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:17.295526 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.391336 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.393054 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.436493 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:17.475005 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.478159 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.603734 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:17.628774 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:17.635648 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:17.680197 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:17.691511 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:17.705127 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:17.715431 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:17.740761 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:17.764317 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:17.775697 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:17.813702 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.820891 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:17.828814 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:17.841244 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:17.861944 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:17.870239 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:17.872295 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:17.913770 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:17.916611 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:17.976544 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.019673 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:18.063887 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:18.075439 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240502 08:21:18.081893 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:18.106237 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:18.118825 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:21:18.121245 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:21:18.121619 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:21:18.136368 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.166560 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.177453 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:18.207772 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:18.212042 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.228608 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:21:18.231570 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:18.255601 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.256565 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.322046 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:18.350996 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:18.366129 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:18.385859 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:21:18.402650 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:18.404130 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:18.438997 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.449687 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:18.462807 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.474575 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.539067 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:18.541512 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:18.563011 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.571105 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:18.580796 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:18.582859 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:18.614856 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.616802 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:18.661154 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:18.736322 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:18.863770 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:18.875043 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.906988 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:18.909225 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:18.956748 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:18.957720 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:19.022642 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:19.027705 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:21:19.035773 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:19.057986 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:19.102593 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:19.119191 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:19.135707 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:19.177760 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:19.179319 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:19.238245 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:19.261585 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:21:19.262665 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:19.263411 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:19.316072 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:19.585249 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:19.598812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:19.605435 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:19.611012 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:19.649030 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:19.667044 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:19.684931 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:19.685658 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:19.743948 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:19.808312 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:19.810591 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:19.812609 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:19.831761 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:19.837345 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:19.841887 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:19.885418 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:19.945605 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.047914 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:20.060495 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.080813 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:20.083233 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.132254 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:20.154742 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:20.160771 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.169600 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:20.194625 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:20.195317 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:20.218920 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.224444 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.263394 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:20.268237 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.282521 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.291951 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:20.302179 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:20.347390 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:20.347921 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.366768 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:20.393612 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:20.422663 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:20.489420 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.551967 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.552160 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.581398 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.657297 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:20.782317 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:20.835423 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:20.852926 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:20.853907 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:20.879632 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:20.899577 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:20.905895 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:20.955535 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:20.965380 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 49198.6us
I20240502 08:21:20.966884 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:20.971118 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:20.979506 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:20.990017 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:21.000349 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:21.023224 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:21.046537 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:21.046864 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:21.090724 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:21.102171 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:21.102717 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:21.125577 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:21.140749 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:21.141341 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:21.157097 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:21.236779 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:21.238937 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:21.255920 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 118121us
W20240502 08:21:21.260046 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:21.298128 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:21.357131 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:21.369339 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:21.501044 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:21.605859 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:21.626456 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:21:21.638303 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:21.658777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:21.665364 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:21.674320 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:21.691989 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:21.748013 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:21.762445 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:21.806690 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:21.826848 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:21.865984 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:21.890517 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:21:21.898499 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:21.908892 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:21.920910 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:22.021386 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:22.105357 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:22.125736 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:22.155200 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:22.160372 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:22.238358 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:22.278486 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.279541 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.353199 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:22.367200 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:22.373495 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 61819.2us
W20240502 08:21:22.421298 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.549456 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.549870 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.571229 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.601621 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:22.674793 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:22.747462 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.750288 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:22.754417 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:22.765287 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:22.767333 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.777781 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:22.786388 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:22.786620 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:22.841311 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:22.865072 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.887408 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.906484 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:22.915485 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:22.927009 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:22.941506 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.948282 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:22.961918 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:22.964324 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:22.970749 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:23.021121 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:23.120299 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:21:23.123617 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.158759 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:21:23.189005 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:21:23.241508 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 08:21:23.242573 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:23.257884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.291571 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:23.306353 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:23.335335 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:23.360190 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:23.383095 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.393360 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:21:23.400790 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:23.404227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.415580 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:23.436304 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:21:23.526726 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:23.574677 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.619055 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.632719 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.634346 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:23.641085 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:23.673928 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.713901 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:23.740370 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:23.772518 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.800269 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:23.819739 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:23.847147 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:23.848722 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:23.859112 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:23.862377 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.867574 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:23.895320 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:23.898988 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:23.980223 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:24.015959 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:24.069386 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:24.141803 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:21:24.157163 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:24.175482 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:24.186638 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:24.223996 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:24.253913 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:24.277179 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:24.298478 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:24.311041 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:24.466686 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:24.469928 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:24.472083 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:24.492225 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:24.499437 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:24.530932 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:24.566996 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:24.606034 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:21:24.680640 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:24.696334 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:24.711419 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:24.776786 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:24.865976 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:24.894753 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:24.899801 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:24.920876 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:24.937966 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.035414 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.065526 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.092388 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:25.118710 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:25.145781 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:25.155853 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:25.184628 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:25.185415 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:25.206945 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:25.232978 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:25.261541 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:25.277137 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.287047 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.322657 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.334080 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.345855 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.384862 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:25.398202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:25.416719 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:25.422546 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:25.433277 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:25.449327 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:25.462324 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:25.491076 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:25.514027 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:25.533608 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.584352 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:25.719523 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:25.727629 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:25.742982 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:25.809935 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:25.824615 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:25.825240 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:25.896257 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:25.902958 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:25.923719 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:25.958665 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:26.004614 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:26.125932 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:26.182178 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.191345 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:26.199757 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:26.203500 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.228430 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:26.233336 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:26.254766 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.272405 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:26.279484 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:26.327432 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:26.346904 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:26.384177 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:26.395964 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.432837 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:26.466387 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:26.508354 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:26.569784 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:26.586895 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.619527 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:26.630736 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:21:26.646157 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:26.671229 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:26.723555 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:26.727439 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:26.761601 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:26.818832 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:26.839527 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:26.861943 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:26.889083 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:26.889633 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:27.086617 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:27.095372 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:27.127933 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:27.139020 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:27.153913 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:27.170794 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:27.210104 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:27.242552 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:21:27.305636 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:27.339118 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:27.342927 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:27.346323 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.357700 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:27.400408 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:27.401331 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:27.442433 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:27.477460 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:27.505842 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:27.531468 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.543555 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.628476 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:27.671675 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.688834 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:27.725911 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.770493 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:27.823534 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:27.854707 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:27.869540 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:27.877985 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:27.907984 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:27.914217 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:27.961470 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:27.964179 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:27.978809 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:28.048152 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:28.053864 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.062721 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.104825 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:28.240242 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:28.244053 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:21:28.246673 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:28.269688 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240502 08:21:28.297761 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:28.303288 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:28.312458 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:21:28.325508 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:28.329351 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:28.340972 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:28.349072 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:28.383268 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:28.397671 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:28.420310 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:28.454825 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:28.476811 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.477016 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.500828 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.501039 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:21:28.508126 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:28.509769 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:28.515892 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:28.525044 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:28.532482 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:21:28.604872 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:28.644529 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:28.653676 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:28.714329 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:28.730527 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:28.730904 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:28.737727 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:28.769634 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.808557 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:21:28.836722 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.847683 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:28.858480 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:28.868805 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:28.876950 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:28.913090 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:28.941184 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:28.941368 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:28.942464 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:28.953892 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:28.979363 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:29.092049 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:29.274960 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:29.300243 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:21:29.337805 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:29.363799 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:29.364629 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:29.371191 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:29.405786 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:29.412804 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:29.427556 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:29.474874 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:21:29.573983 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:29.770857 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:29.807745 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:29.836493 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:29.843210 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:29.870177 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:29.870927 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:29.886140 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:29.894697 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:29.907734 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:29.910492 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:29.921406 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:29.932658 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:29.940799 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:29.951913 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:29.958947 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:29.989732 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:29.991395 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:30.071408 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:30.168794 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:30.250676 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.263415 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:30.285452 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:30.289916 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.321864 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:30.330816 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:30.330971 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:30.340731 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:30.352592 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.375474 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:30.378017 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.393558 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:30.410744 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.431917 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:30.435660 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:30.443169 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:30.485677 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:30.509929 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:30.520145 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:30.614567 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:30.643013 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.712841 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:30.738806 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:30.744946 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.753109 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:30.768075 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:30.779886 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:30.782298 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:30.803995 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.836699 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:30.887491 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.942984 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:30.966848 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:30.998307 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:31.010332 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:31.081053 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:31.084192 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:31.088255 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:31.092607 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:31.138029 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:31.194795 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:31.197546 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:31.223147 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:31.236109 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:31.243616 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:31.271703 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:31.272037 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:31.319340 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:31.355557 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:31.371536 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:31.388793 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:31.401124 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:21:31.417333 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:31.441567 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:31.620525 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:31.659116 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:31.723526 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:31.781404 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:31.799996 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:31.808343 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:31.853719 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:31.881235 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:31.898384 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:31.920001 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.012207 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.039913 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.041148 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:32.064522 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:32.100153 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:32.127125 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 53822.6us
W20240502 08:21:32.131824 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:32.140724 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:32.193859 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:21:32.263053 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:32.296265 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:32.369844 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:32.376742 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.393011 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:32.414180 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:32.424049 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:32.433912 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.472890 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:32.555125 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:32.583792 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:32.596210 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:32.640506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.668007 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:32.671752 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:32.688020 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:32.690622 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:32.734210 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:32.753830 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:32.755872 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:32.757689 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.781402 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:32.785567 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:32.822157 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:32.829026 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:32.840679 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:32.842733 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:32.849197 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:32.888146 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:32.896898 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.076900 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:33.080368 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.088677 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.092667 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.118049 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.231030 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.240371 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:33.256282 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 08:21:33.260651 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.369266 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:33.379289 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 08:21:33.404826 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:33.417071 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:33.424412 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:33.429894 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:33.432904 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:33.450989 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:21:33.452183 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.472426 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:33.477464 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.488559 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:33.512405 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:33.552258 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:33.556921 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:33.585028 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:33.607677 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:21:33.648666 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.678356 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:33.704281 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:21:33.707839 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:33.713356 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:33.730346 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.732192 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:33.733690 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.759109 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:33.788554 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:33.809590 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:33.843169 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:33.848672 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:33.864120 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:33.903594 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:33.936611 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:33.972963 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:33.979089 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:33.994081 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:21:34.002074 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:34.002519 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:21:34.024292 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:34.089576 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:34.092878 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:34.115124 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:34.192554 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:34.432011 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:34.437664 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:34.573439 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:34.585780 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:34.605116 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:34.609618 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:34.666271 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:34.679766 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:34.737474 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:21:34.788856 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:34.860170 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:34.861225 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:34.874910 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:34.892971 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:34.921597 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:34.963559 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:34.966769 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:35.080035 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:35.086194 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.129956 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.144968 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:35.184847 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:35.211686 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:35.236701 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:35.239681 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:35.260176 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:35.289011 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:35.289458 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:35.291507 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:35.302400 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.404320 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.428231 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:35.461701 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:35.489590 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:35.511366 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:35.534325 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.555804 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:35.562800 21793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:35.562950 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:35.564561 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:35.586910 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:35.590854 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:35.599726 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:35.625229 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:35.640466 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.641535 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:35.646387 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:35.661787 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:35.680284 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:35.706570 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:35.796631 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.816893 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:35.846935 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:35.847357 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.037882 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:36.042259 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:36.100219 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.100417 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:36.101356 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:36.139734 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.187861 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:36.195115 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:36.210839 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:36.217291 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.273840 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:36.297339 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:36.328161 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.343155 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:36.365013 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.367417 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:36.393575 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:36.404693 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:36.421825 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:36.428606 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:36.464267 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:36.493265 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.610251 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:36.645905 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:36.670038 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:36.683871 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:21:36.697633 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:36.754290 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:36.756740 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:36.784734 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:36.802573 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:36.948587 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:37.002262 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:37.015952 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:37.034363 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:37.036927 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:21:37.043784 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.080230 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:37.097498 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:37.290266 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.291311 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:37.317624 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:37.320259 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:37.351672 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.362915 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:37.366796 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:37.395953 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:37.407596 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:37.443636 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:37.455410 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:37.509191 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:37.562047 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.733336 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:37.765825 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:37.782363 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:37.792006 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:37.833349 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.835709 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:37.859762 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:37.866277 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:37.961869 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.972110 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:37.973802 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:37.977921 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:37.994787 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:38.004089 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:38.004407 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:38.006762 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:38.010797 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:38.020399 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:38.023736 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.037617 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.081976 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.178104 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.208568 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:38.222239 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:38.250901 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:38.259121 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 08:21:38.259523 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:38.260228 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:38.267527 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:38.317864 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:38.329170 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:38.373759 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:38.379513 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:38.386144 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:38.389175 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:38.410035 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:38.438916 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:38.443475 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:38.469084 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:38.484380 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.489818 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:21:38.516502 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.570389 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:21:38.641371 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:38.642030 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:38.655130 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.680032 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.687927 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:38.688239 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:38.705116 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:38.712714 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:21:38.718819 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:21:38.720289 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:38.720469 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:38.722651 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:38.723767 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:38.781211 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:38.783385 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:38.984771 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:39.007391 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:21:39.046403 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:21:39.143805 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:39.316339 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:39.367514 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:39.368340 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:39.373492 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:39.394685 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.396121 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.423017 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.435331 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:39.447202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.497489 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:39.528051 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:21:39.528409 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:39.541723 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:39.545419 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:39.573550 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:39.591414 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:39.594337 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:39.595037 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:39.598527 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:39.604367 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:39.618989 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:39.622977 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:39.642932 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 68524.2us
I20240502 08:21:39.659310 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.679960 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:39.695832 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:39.698271 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.730964 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:39.735322 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:39.756243 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:39.830299 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:39.865768 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:39.888202 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:40.042060 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.071658 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:40.089054 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:40.097587 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.130380 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:40.148270 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:40.151337 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.173980 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:40.234946 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:40.256049 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:40.303730 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:40.446331 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:40.475286 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:40.526299 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:40.542843 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:40.560612 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.614953 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:40.637596 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:40.684497 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:40.685515 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:40.730849 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.789613 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:40.823482 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.830205 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:40.837956 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 53264.5us
W20240502 08:21:40.849079 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:40.860388 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:40.864455 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:40.930176 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:41.025959 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:41.036659 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:41.126242 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:41.175806 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:41.183658 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.213068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:41.213068 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:41.217634 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.230942 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:41.256836 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:41.286581 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:41.295109 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 73318.7us
W20240502 08:21:41.295856 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.311947 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:41.338789 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:41.345829 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:41.347637 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:41.349165 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:41.349933 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:41.362557 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.377301 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:41.387485 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:41.419353 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.421064 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:41.421562 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:41.447471 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:41.478016 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:41.481391 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.483314 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:41.529616 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:41.573773 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:41.716149 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:21:41.735837 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 69145us
W20240502 08:21:41.808749 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:41.899873 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:41.930219 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:41.944098 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:41.950479 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:41.978091 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:42.002664 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:42.020814 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:42.027870 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:21:42.037330 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:42.054983 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:42.159082 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:42.189303 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:42.194317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:42.232719 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:42.233117 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:42.261456 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:42.282630 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:42.411752 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:42.423951 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:42.452541 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:42.568025 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:42.670650 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:42.673825 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:42.694028 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:42.736303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:42.739656 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:42.742143 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:42.769860 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:42.770795 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:42.787883 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:42.793040 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:42.797273 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:42.800371 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:42.803668 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:42.816298 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:42.853742 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:42.856045 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:42.879467 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:42.880896 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:42.938977 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 49679.1us
W20240502 08:21:42.947751 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:42.948872 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:42.960959 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:42.985523 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:42.990137 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:43.034682 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.036828 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.111446 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.168258 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.174484 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.176412 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.189172 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:43.191349 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:43.242069 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:43.270668 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:43.290032 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 08:21:43.309226 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:43.322856 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:43.324013 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:43.339398 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:43.388957 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:43.395491 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:43.454736 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.515983 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:21:43.522915 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.533654 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.545643 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:43.545723 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:43.569473 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:43.602031 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:43.623806 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:21:43.625152 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:43.675967 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:43.678526 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:43.679009 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:43.748174 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:43.769639 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.789333 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:21:43.827296 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:43.896140 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:43.922469 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:21:43.923574 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:43.949414 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:43.980715 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:44.039919 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:44.071791 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:44.089548 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:44.123492 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:44.133050 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:44.210412 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:21:44.316954 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:44.378473 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:44.411098 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:44.418092 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:44.428750 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:44.436141 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:21:44.468346 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:44.481314 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:44.493103 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:44.509835 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:44.512893 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 76755us
I20240502 08:21:44.534833 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:44.584918 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:44.594030 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:44.611900 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:44.617652 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:44.627004 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:44.650513 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:44.664947 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:44.684319 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:21:44.688633 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:44.713405 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:44.714097 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:44.754354 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:44.761808 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:44.802919 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:44.851382 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:44.914888 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:44.931187 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.072053 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:45.149845 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:45.180523 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:45.191985 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:45.192973 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:45.219856 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.240116 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.280699 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:45.300269 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:45.300889 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:45.321920 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:45.345168 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.368949 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:45.396371 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:45.413080 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:45.471088 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.496726 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:45.502179 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:45.534340 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.552722 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:45.569715 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.699930 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.724961 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.754108 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.764228 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:45.782752 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.795601 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:45.804323 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.818426 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:45.846287 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:45.860971 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:45.894194 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:45.898963 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:45.904381 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:45.915850 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:45.966311 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:45.981441 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:46.002519 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:46.023715 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 45940us
W20240502 08:21:46.037739 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:46.070667 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:46.081606 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:46.113726 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:46.117270 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:46.213625 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:46.214432 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:46.224249 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:46.250804 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:46.390614 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:46.440898 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:46.456245 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:46.488420 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:46.492890 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:46.494233 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:46.521287 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:46.527907 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:46.543061 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:46.595315 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:46.721251 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:46.745122 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:46.794584 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:46.798914 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:46.830827 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:46.855917 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:46.864902 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:46.881148 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:46.954114 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:46.999287 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:47.054816 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.063282 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:47.117090 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:21:47.117259 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:21:47.122619 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:47.134869 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:47.145808 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:47.147348 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:47.164412 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:47.190503 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:47.194216 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:47.205520 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.215029 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:47.216606 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:47.230094 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:47.241817 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.247519 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:47.356873 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:47.532730 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:47.562016 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.630051 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:47.676983 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:47.727591 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:47.749995 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.777434 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:47.820405 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:47.844019 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:47.871646 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:47.922937 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:47.940135 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:47.945952 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:47.975797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:47.993136 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:48.044195 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:48.061748 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:48.064209 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:48.077678 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:48.084247 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:48.119760 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:48.128680 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:48.146554 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:48.159119 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.169785 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.220041 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.235880 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.315548 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.344801 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.355890 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.389566 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 08:21:48.390399 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:48.390831 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:48.405717 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:48.441524 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:48.455461 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:48.466408 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:48.475801 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:48.480629 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:48.500609 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:48.501863 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:48.549429 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:48.554898 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:48.565333 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:21:48.576638 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:48.587306 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:48.593434 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:48.779613 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:48.812243 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:21:48.813400 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:48.855304 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:48.860296 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:48.882133 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:48.889881 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:48.924754 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:48.925783 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:48.942895 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:48.968587 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:49.121522 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:49.155087 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 08:21:49.188719 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:49.191805 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:21:49.241530 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:49.271332 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:49.279232 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:49.297871 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:49.320129 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:49.321105 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:49.368641 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:49.373979 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:49.454074 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:49.476282 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:49.494518 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:49.496923 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:49.514096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:49.538398 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:49.547968 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:49.560986 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:49.560981 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:49.569365 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:49.575667 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:49.577801 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:21:49.581374 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:49.605149 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:21:49.613785 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:49.628096 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:21:49.630170 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:49.650245 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:49.666177 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:49.698344 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:49.890967 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:49.892194 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:49.914446 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:49.931788 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:49.942736 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:49.958603 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:49.967329 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:50.082440 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:50.161811 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.271704 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 71348.9us
I20240502 08:21:50.295383 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:50.311630 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:50.331404 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:50.337000 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:50.339553 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:50.352321 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.405380 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:50.406437 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:50.427968 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:50.451270 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.460100 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:50.523818 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.546730 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.556598 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.603001 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.626138 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.666635 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:50.671711 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:50.730351 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:50.731848 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:50.758200 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:50.759851 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:50.767195 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.781252 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.785753 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.794358 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.812795 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:50.822598 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:50.949092 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:50.960669 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:50.973752 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:51.079900 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:51.150805 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:51.153195 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:51.164340 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:51.186038 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:51.194453 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:51.286530 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:51.292309 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033398
I20240502 08:21:51.301319 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:51.310137 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:51.372480 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:51.397322 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:51.397601 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:51.398772 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:51.400424 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:51.413096 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:51.429179 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:51.436601 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:51.438021 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:51.443763 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:51.447899 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:51.492935 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:51.510545 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:51.544140 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:51.622076 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.302s user 0.051s sys 0.005s Metrics: {"bytes_written":20528,"cfile_init":1,"dirs.queue_time_us":65993,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1611,"drs_written":1,"lbm_read_time_us":4557,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1264,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":130100,"threads_started":2,"wal-append.queue_time_us":51945}
I20240502 08:21:51.623761 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7021 bytes on disk
I20240502 08:21:51.631235 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240502 08:21:51.653550 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:51.748579 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:51.792555 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:51.805497 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:51.862946 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:51.891788 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:21:51.909435 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:51.934638 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:51.937008 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:52.003053 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:52.010919 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:52.046270 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:52.056294 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:52.120163 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:52.161494 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:52.223984 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:21:52.352722 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:52.358949 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:52.375800 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:52.382810 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:52.411154 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:52.417977 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:52.440995 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:52.442961 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:52.462117 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:52.488446 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:52.533282 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:52.561002 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:52.601505 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:52.677804 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:52.723644 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:52.773464 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:52.796640 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:52.797394 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:52.808424 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:52.821765 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:52.824506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:52.861070 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:52.862368 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:52.870116 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:52.924510 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:52.970312 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:52.971460 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:52.992110 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.027662 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:53.031162 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.063115 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:53.068322 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.087934 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:53.093591 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:53.116714 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:53.154214 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.165305 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:53.215276 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.233495 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.293000 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.391534 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:53.406976 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:53.406951 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.413292 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 08:21:53.429519 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:53.456007 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:53.478978 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:53.508237 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:53.509588 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:53.515910 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.528622 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.531352 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:53.537448 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:53.557621 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:53.568576 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:53.573484 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:53.575086 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:53.604846 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:53.609025 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:53.675755 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 08:21:53.844116 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:21:53.845319 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.859119 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:53.879895 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:53.915913 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:54.045715 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:54.051548 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:54.098621 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:54.141032 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:54.181254 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:54.185245 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:54.240885 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:54.294349 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:54.312522 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:54.317868 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:21:54.372692 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:21:54.403606 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:54.420423 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:54.496157 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:54.513371 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:21:54.514514 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:54.515363 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:54.525570 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:54.527820 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:54.586062 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:54.596745 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:54.599917 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:54.619288 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:21:54.619980 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:54.626967 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:54.628664 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:21:54.646404 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:54.672189 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:54.679052 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:54.699558 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:54.700189 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:54.733894 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:54.758262 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:54.791147 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:54.796764 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:54.800662 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:54.815814 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:54.819406 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:54.831707 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:55.034492 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:55.100425 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:55.103960 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:55.147581 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:55.179134 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.181690 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:55.215131 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.257247 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:55.257794 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:55.286697 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.311372 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:55.317090 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.323319 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:55.335655 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:55.344058 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.349437 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:55.372123 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:55.402628 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.405164 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:55.431176 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:55.442122 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.547860 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.632650 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.639631 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.647553 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:55.734269 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.796371 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.855346 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:55.886214 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:55.910248 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:55.916045 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:55.947975 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:56.005048 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:56.018209 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:56.067312 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:56.071432 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:56.100756 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:56.107756 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:56.111307 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:56.133387 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:56.136477 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:56.166666 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:56.168879 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:56.177975 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:56.188140 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:56.193419 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:56.200712 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:56.227077 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:56.229122 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:56.246922 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:56.265117 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:56.266168 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:56.275518 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:56.280499 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:56.295688 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:56.322702 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:56.337502 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:56.370159 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:56.387650 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:56.441133 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:56.451856 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:56.483426 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:56.658888 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:56.671209 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:56.683985 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:56.684379 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:56.713706 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:56.723001 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:56.726851 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:56.755059 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:21:56.764664 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:56.807307 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:56.851853 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 62976.2us
W20240502 08:21:56.898985 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:56.926398 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:57.045210 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:57.060957 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:57.102914 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:57.107970 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:57.139385 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:57.160794 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:57.207715 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:57.218797 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:57.234212 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:21:57.256012 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.362192 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:57.362512 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:57.382050 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.390836 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:57.473184 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:57.483686 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:57.504850 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:57.538125 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:57.597450 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:57.617467 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.657145 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:57.686507 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:57.693444 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.716492 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:57.755123 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:57.772686 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.774295 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:57.774751 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:57.788933 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:57.800882 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:57.821110 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:57.823554 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:57.851614 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:57.862040 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:57.905968 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:57.948259 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.010116 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:21:58.138446 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.148389 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:58.201159 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.203698 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:58.204427 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:58.248780 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:58.250108 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:58.262054 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.267923 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.277576 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:58.287757 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:58.309975 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:58.323494 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:58.345155 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:58.481494 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:58.484690 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:58.489187 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:58.491567 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:21:58.492518 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:58.502749 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:58.520449 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:58.540812 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:58.549350 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:58.553452 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.607326 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.619045 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:58.733872 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 08:21:58.766243 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:58.811538 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:21:58.844712 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:21:58.849074 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:58.849120 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:21:58.891671 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:21:58.935124 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:21:58.945461 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:58.946940 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:58.969980 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:58.978056 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:59.096159 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:21:59.097285 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:59.157492 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:21:59.202243 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:59.203859 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:59.208544 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:59.223470 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:21:59.236040 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:59.285326 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:21:59.317476 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:59.324493 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:59.326817 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:59.489507 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:59.540197 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:21:59.559536 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:59.610376 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(d4d933eacd944d84b39f0cb9c45641f8): perf score=0.033343
W20240502 08:21:59.644073 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:21:59.653550 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e): perf score=0.033364
I20240502 08:21:59.682067 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf): perf score=0.033369
W20240502 08:21:59.685597 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:21:59.716071 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:21:59.735112 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:21:59.768937 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:21:59.796191 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:21:59.826697 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:21:59.834442 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:21:59.843215 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:21:59.868954 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:21:59.921204 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:21:59.939356 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:21:59.994553 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:21:59.998797 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:00.004503 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:00.016265 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:00.041188 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:00.043939 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:00.046176 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:00.079710 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:00.091583 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:22:00.097414 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:00.105358 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 52858.1us
W20240502 08:22:00.110343 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:00.118692 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:00.129896 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:00.152478 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:00.159334 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:00.166296 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:00.195065 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:00.233806 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:00.263149 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:00.279337 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:00.302245 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:00.323940 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:00.367092 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:00.395364 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:00.428630 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.460511 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 135394us
I20240502 08:22:00.464648 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.810s user 0.081s sys 0.051s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":40434,"dirs.queue_time_us":32755,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":11791,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":185179,"threads_started":3,"wal-append.queue_time_us":107057}
W20240502 08:22:00.465767 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:00.468355 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e): 4840 bytes on disk
I20240502 08:22:00.469254 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 08:22:00.470431 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf): perf score=0.033588
W20240502 08:22:00.511166 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.513813 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.542608 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:00.547694 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:00.635450 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(d4d933eacd944d84b39f0cb9c45641f8) complete. Timing: real 1.024s user 0.065s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":489,"dirs.queue_time_us":12397,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1121,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1167,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56874,"threads_started":3,"wal-append.queue_time_us":27885}
I20240502 08:22:00.644932 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8): 4925 bytes on disk
I20240502 08:22:00.646095 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 08:22:00.690549 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 1.008s user 0.086s sys 0.036s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":22568,"dirs.queue_time_us":168352,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":17061,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":971,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":222446,"threads_started":3,"wal-append.queue_time_us":13433}
I20240502 08:22:00.692788 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf): 4586 bytes on disk
I20240502 08:22:00.699163 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 08:22:00.703341 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 0.233s user 0.044s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12817,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":8683,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":44230,"threads_started":2,"wal-append.queue_time_us":20930}
I20240502 08:22:00.704751 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf): 4586 bytes on disk
I20240502 08:22:00.705471 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":168,"lbm_reads_lt_1ms":4}
W20240502 08:22:00.823971 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.859773 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:00.871786 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.956049 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:00.966691 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:01.043311 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:01.071635 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:01.153110 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.170688 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:01.244936 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.248322 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:01.258301 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:01.276046 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:01.299060 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:01.364715 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.384802 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:01.414335 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:01.458030 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:22:01.553711 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:01.605386 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.606902 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:01.607173 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:01.678172 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:01.685631 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.692068 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:01.696058 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:01.718164 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.723421 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:01.739964 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:01.748088 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:01.762881 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:01.763761 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:01.777680 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:01.808022 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:01.809465 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:01.828333 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:01.853359 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:01.902736 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2): perf score=0.033364
I20240502 08:22:01.962589 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(2de41d6c6de0400397645c05a0fa17de): perf score=0.033378
I20240502 08:22:01.965255 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2): perf score=0.033376
W20240502 08:22:01.980005 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:01.988411 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(2de41d6c6de0400397645c05a0fa17de): perf score=0.033387
W20240502 08:22:02.132860 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:02.114050 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.151s user 0.038s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":21243,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":7648,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":52021,"threads_started":2,"wal-append.queue_time_us":12825}
I20240502 08:22:02.134629 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(dc49c050280346db88d1323e30513c80): perf score=0.033423
W20240502 08:22:02.154213 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:02.165784 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:02.168269 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:02.213862 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:02.217307 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:02.240906 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:02.263630 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:02.327953 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:02.330808 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:22:02.342551 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:02.344785 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:02.367450 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:02.370918 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:02.414407 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:02.423615 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:02.440330 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:02.460040 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:02.463852 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(dc49c050280346db88d1323e30513c80) complete. Timing: real 0.327s user 0.039s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1587,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":50020,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19608,"threads_started":1,"wal-append.queue_time_us":19491}
I20240502 08:22:02.465242 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de): 4925 bytes on disk
I20240502 08:22:02.458935 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.470s user 0.061s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":16619,"dirs.queue_time_us":16867,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1734,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41214,"threads_started":3,"wal-append.queue_time_us":11374}
I20240502 08:22:02.466495 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4}
I20240502 08:22:02.467116 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de): 4925 bytes on disk
I20240502 08:22:02.467777 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(dc49c050280346db88d1323e30513c80): 4586 bytes on disk
I20240502 08:22:02.471836 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(dc49c050280346db88d1323e30513c80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 08:22:02.472942 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240502 08:22:02.485338 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:02.540529 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.637s user 0.053s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":358,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":2363,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11805,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123316,"threads_started":2,"wal-append.queue_time_us":102073}
I20240502 08:22:02.542325 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2): 4838 bytes on disk
I20240502 08:22:02.558936 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.593s user 0.068s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":27023,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":4509,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1084,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33900,"threads_started":2,"wal-append.queue_time_us":6611}
I20240502 08:22:02.560497 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2): 4838 bytes on disk
I20240502 08:22:02.561118 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 08:22:02.563835 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 08:22:02.649856 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:02.664188 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:02.681098 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:02.722843 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:02.833907 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:02.895635 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:02.967104 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.110077 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:03.150025 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.172551 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:03.244941 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.287359 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:03.288378 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.323769 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:03.330263 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:03.339509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.342053 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:03.362426 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:03.394544 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:03.400859 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:03.410490 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 82390.7us
I20240502 08:22:03.414572 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:03.415117 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:03.429759 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:03.430719 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.441401 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.442898 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:03.494077 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:03.510648 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 08:22:03.547366 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.552501 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:03.592931 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:03.625231 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:03.628816 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:03.630498 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.637238 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:03.712936 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.761332 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:03.796635 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:22:03.823138 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.860566 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:03.902906 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:03.974246 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:03.983331 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(80b3ba1d834a4314b63f336675845441): perf score=0.033334
I20240502 08:22:03.997131 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:04.017779 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:04.072520 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): perf score=0.033360
I20240502 08:22:04.089120 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:04.091382 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:04.115897 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:04.121692 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:22:04.144165 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:04.160022 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:04.181664 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:04.199702 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:04.231521 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(80b3ba1d834a4314b63f336675845441): perf score=0.033397
W20240502 08:22:04.260610 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:04.328047 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.255s user 0.039s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":19083,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":4120,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3280,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69217,"threads_started":2,"wal-append.queue_time_us":50051}
I20240502 08:22:04.330236 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): 4924 bytes on disk
I20240502 08:22:04.340183 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:04.341183 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 08:22:04.344388 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(a20b2fd9f1ce488ba7793ed7ac268c16): perf score=0.033429
W20240502 08:22:04.419405 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:04.447163 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.464s user 0.051s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":86899,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":202,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1113,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":172868,"threads_started":2,"wal-append.queue_time_us":11778}
W20240502 08:22:04.448704 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:04.456455 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441): 4587 bytes on disk
I20240502 08:22:04.457784 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 08:22:04.533329 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(a20b2fd9f1ce488ba7793ed7ac268c16) complete. Timing: real 0.188s user 0.066s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1470,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12600,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2116,"threads_started":1,"wal-append.queue_time_us":12031}
I20240502 08:22:04.535427 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(a20b2fd9f1ce488ba7793ed7ac268c16): 4840 bytes on disk
I20240502 08:22:04.543921 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(a20b2fd9f1ce488ba7793ed7ac268c16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240502 08:22:04.568873 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:22:04.600495 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:22:04.628008 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:04.636559 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:04.670157 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:04.727483 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:22:04.753422 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.522s user 0.053s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26678,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":3021,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":985,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":190147,"threads_started":2,"wal-append.queue_time_us":160562}
I20240502 08:22:04.756346 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441): 4587 bytes on disk
W20240502 08:22:04.759896 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:04.762204 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:04.782369 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:22:04.786868 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.024s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":23338,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2}
W20240502 08:22:04.816097 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:04.820063 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:04.849031 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:04.867398 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:04.891849 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:04.931706 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:04.959138 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:04.971621 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:04.997983 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:05.013634 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:05.030268 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.033596 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:05.053390 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:05.076371 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:05.079952 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:05.082787 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:05.092118 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:05.107836 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.108747 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:05.117159 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:05.123589 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:05.148011 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.214849 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(296ccf9be99545789bbb3ff4031d3170): perf score=0.033346
W20240502 08:22:05.225631 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:05.255618 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:22:05.256614 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.290755 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(cec56facd7f740aeb42bc7c32495dae3): perf score=0.033371
I20240502 08:22:05.298718 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80): perf score=0.033372
I20240502 08:22:05.324935 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(296ccf9be99545789bbb3ff4031d3170): perf score=0.033382
I20240502 08:22:05.336370 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(296ccf9be99545789bbb3ff4031d3170) complete. Timing: real 0.121s user 0.034s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6577,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":22091,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":946,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17118,"threads_started":2,"wal-append.queue_time_us":10227}
I20240502 08:22:05.340378 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170): 4586 bytes on disk
I20240502 08:22:05.342128 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 08:22:05.410413 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:05.445304 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:05.489112 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:05.497023 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:05.521121 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:05.544263 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:05.583039 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(296ccf9be99545789bbb3ff4031d3170) complete. Timing: real 0.258s user 0.120s sys 0.101s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6404,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":12505,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":182416,"threads_started":2,"wal-append.queue_time_us":176031}
I20240502 08:22:05.584514 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80): perf score=0.033448
W20240502 08:22:05.588012 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:05.588331 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.289s user 0.105s sys 0.102s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":92682,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":403840,"thread_start_us":222119,"threads_started":2,"wal-append.queue_time_us":128672}
I20240502 08:22:05.590165 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80): 4925 bytes on disk
I20240502 08:22:05.590790 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240502 08:22:05.609762 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.685703 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.395s user 0.055s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":21326,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":5351,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1594,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":41711,"threads_started":2,"wal-append.queue_time_us":6038}
I20240502 08:22:05.690065 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3): 4840 bytes on disk
I20240502 08:22:05.691155 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240502 08:22:05.724362 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.746416 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.158s user 0.043s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":319,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1503,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4000,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15009,"threads_started":1,"wal-append.queue_time_us":15175}
I20240502 08:22:05.748095 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80): 4925 bytes on disk
W20240502 08:22:05.748286 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.749401 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 08:22:05.751544 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170): 4586 bytes on disk
I20240502 08:22:05.752132 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 08:22:05.817255 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:05.817412 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:05.817497 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:05.857246 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:05.961849 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:05.980067 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:05.991644 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:06.014518 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:06.042330 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:06.062991 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:06.081918 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:06.138046 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:06.141185 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:06.170892 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:06.208802 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:06.265974 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:06.279189 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.318478 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:06.345170 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68): perf score=0.033350
I20240502 08:22:06.360409 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:06.417974 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:06.446194 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(1648586b889b4a1e9ac34481cec6257a): perf score=0.033355
I20240502 08:22:06.446815 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:06.462972 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:06.511174 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(59f5d97e6e0a4755be41699246907e55): perf score=0.033371
W20240502 08:22:06.554443 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:06.560118 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:06.590606 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.596447 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.604933 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68): perf score=0.033397
W20240502 08:22:06.611157 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:06.622020 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:06.631420 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:06.645303 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:06.652081 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:06.682243 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:06.700379 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:06.719856 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.372s user 0.050s sys 0.027s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39655,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":25944,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":63147,"threads_started":2,"wal-append.queue_time_us":22997}
I20240502 08:22:06.721226 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68): 4925 bytes on disk
I20240502 08:22:06.722060 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 08:22:06.723162 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(1648586b889b4a1e9ac34481cec6257a): perf score=0.033449
I20240502 08:22:06.760432 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:06.774863 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:06.777839 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:06.810240 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.823073 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.862507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:06.862591 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:06.878516 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:06.923379 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.199s user 0.071s sys 0.058s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1107,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137449,"threads_started":1,"wal-append.queue_time_us":137549}
I20240502 08:22:06.924981 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a): 4586 bytes on disk
I20240502 08:22:06.929383 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:06.943904 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240502 08:22:06.958959 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:06.994115 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:07.009431 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(59f5d97e6e0a4755be41699246907e55) complete. Timing: real 0.498s user 0.040s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":17870,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2796,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":52438,"threads_started":2,"wal-append.queue_time_us":33555}
I20240502 08:22:07.010905 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(59f5d97e6e0a4755be41699246907e55): 4840 bytes on disk
I20240502 08:22:07.017547 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.412s user 0.039s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33605,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":63581,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":88190,"threads_started":2,"wal-append.queue_time_us":53743}
I20240502 08:22:07.018895 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68): 4925 bytes on disk
I20240502 08:22:07.019693 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(59f5d97e6e0a4755be41699246907e55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 08:22:07.026504 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 08:22:07.084010 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.637s user 0.061s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39694,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":3707,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":35641,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97015,"threads_started":2,"wal-append.queue_time_us":56522}
I20240502 08:22:07.085901 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a): 4586 bytes on disk
I20240502 08:22:07.087963 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 08:22:07.143190 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:07.176141 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:07.177572 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:22:07.181298 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:07.201295 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:07.217774 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:22:07.240717 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:07.249799 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:07.333987 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:07.366972 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:07.385071 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:07.444104 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:07.461534 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:07.462343 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:07.481187 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:07.487483 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:07.508206 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:07.517004 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:07.530570 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:07.536635 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:07.566676 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:07.672586 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:07.700909 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(b88c4a1a36424086aad89b7585c17960): perf score=0.033346
W20240502 08:22:07.735615 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:07.780690 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(e0241228bd204ca2a8ad2c57b6eeb543): perf score=0.033380
I20240502 08:22:07.784302 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.083s user 0.047s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26414,"dirs.run_cpu_time_us":296,"dirs.run_wall_time_us":3746,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32789,"threads_started":2,"wal-append.queue_time_us":6384}
I20240502 08:22:07.785830 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960): 4586 bytes on disk
W20240502 08:22:07.786803 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:07.802112 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 08:22:07.842444 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(b88c4a1a36424086aad89b7585c17960): perf score=0.033390
W20240502 08:22:07.949699 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.017271 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.033808 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:08.097327 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 52014.8us
W20240502 08:22:08.099483 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.113343 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:08.195351 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:08.200613 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:08.203325 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:08.208637 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:08.234228 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(e0241228bd204ca2a8ad2c57b6eeb543) complete. Timing: real 0.453s user 0.041s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46411,"dirs.run_cpu_time_us":332,"dirs.run_wall_time_us":36726,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77272,"threads_started":2,"wal-append.queue_time_us":30860}
I20240502 08:22:08.235688 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(e0241228bd204ca2a8ad2c57b6eeb543): 4840 bytes on disk
I20240502 08:22:08.238976 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:08.246258 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(e0241228bd204ca2a8ad2c57b6eeb543) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2451,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240502 08:22:08.251689 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.409s user 0.050s sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":151721,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":9207,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":938,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":198058,"threads_started":2,"wal-append.queue_time_us":46104}
I20240502 08:22:08.253721 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960): 4586 bytes on disk
I20240502 08:22:08.260566 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
I20240502 08:22:08.262501 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(07f5c2dfcd974ee2b8720e9eb052c3f7): perf score=0.033502
I20240502 08:22:08.272539 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:08.333124 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.343302 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:08.374560 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:08.387739 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.399395 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:08.418363 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:08.421546 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:08.431371 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:08.519119 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:22:08.526424 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.535498 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.547204 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:08.570379 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.582280 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.585150 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:08.632290 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:08.635648 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:08.649312 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.657218 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:08.657910 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:08.676224 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:08.698649 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(07f5c2dfcd974ee2b8720e9eb052c3f7) complete. Timing: real 0.436s user 0.071s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1023,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":95374,"threads_started":1,"wal-append.queue_time_us":89346}
I20240502 08:22:08.700361 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(76bb1e3e2b9d4e31bf93ac1ce08a7833): perf score=0.033336
I20240502 08:22:08.710732 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:08.718611 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:08.742830 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:08.751838 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:08.756693 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:08.768100 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.791030 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(852613f14b324b38b5eecc29d4c602cd): perf score=0.033368
W20240502 08:22:08.798847 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 08:22:08.809226 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(182a6f2df91242cfa10f05ef788a19cb): perf score=0.033370
W20240502 08:22:08.817644 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.854492 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:08.861739 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:08.896304 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:08.919890 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:22:08.956099 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(76bb1e3e2b9d4e31bf93ac1ce08a7833) complete. Timing: real 0.180s user 0.034s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1045,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6645,"threads_started":1,"wal-append.queue_time_us":6525}
I20240502 08:22:08.964748 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7): 4925 bytes on disk
I20240502 08:22:08.965858 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240502 08:22:08.967826 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833): 4840 bytes on disk
I20240502 08:22:08.968427 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 08:22:09.021031 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:09.041185 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:09.050477 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:09.055404 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:09.059639 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(182a6f2df91242cfa10f05ef788a19cb) complete. Timing: real 0.250s user 0.058s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":69434,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":2882,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":925,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":92658,"threads_started":2,"wal-append.queue_time_us":21798}
I20240502 08:22:09.062263 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(182a6f2df91242cfa10f05ef788a19cb): 4925 bytes on disk
I20240502 08:22:09.062903 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(182a6f2df91242cfa10f05ef788a19cb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 08:22:09.073081 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:09.073498 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:09.085456 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:09.106257 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:09.115711 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:09.224870 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(852613f14b324b38b5eecc29d4c602cd) complete. Timing: real 0.434s user 0.069s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15849,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":5904,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":933,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26793,"threads_started":2,"wal-append.queue_time_us":10947}
I20240502 08:22:09.226315 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(852613f14b324b38b5eecc29d4c602cd): 4584 bytes on disk
I20240502 08:22:09.229050 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(852613f14b324b38b5eecc29d4c602cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 08:22:09.265882 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:09.321923 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:22:09.329754 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:09.364041 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:09.398613 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:09.453845 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:09.487521 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:09.552104 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:09.563625 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:09.564203 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:09.572966 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:09.587376 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:09.602070 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:09.604514 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:09.604971 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:09.677328 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:09.694661 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:09.709663 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:09.728884 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:09.756453 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:22:09.757720 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:09.759567 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:09.845624 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:09.846488 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:09.941727 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:22:09.959769 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:09.974470 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419): perf score=0.033375
I20240502 08:22:09.984401 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(2bfc3eaed583458b8381d4e7b4d36aef): perf score=0.033380
I20240502 08:22:10.006350 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(2bfc3eaed583458b8381d4e7b4d36aef): perf score=0.033378
I20240502 08:22:10.069258 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419): perf score=0.033392
W20240502 08:22:10.133983 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:22:10.135957 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:10.191735 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.217s user 0.064s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":73644,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":2460,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94658,"threads_started":2,"wal-append.queue_time_us":4280}
I20240502 08:22:10.203541 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419): 4840 bytes on disk
I20240502 08:22:10.205035 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
W20240502 08:22:10.206481 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:10.207262 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:10.218371 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(2bfc3eaed583458b8381d4e7b4d36aef) complete. Timing: real 0.212s user 0.078s sys 0.048s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":77460,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":4682,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":864,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":7552,"thread_start_us":81617,"threads_started":2,"wal-append.queue_time_us":4109}
I20240502 08:22:10.219785 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6): perf score=0.033434
I20240502 08:22:10.310644 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:10.335768 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.112s user 0.038s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1153,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7583,"threads_started":1,"wal-append.queue_time_us":7394}
I20240502 08:22:10.338003 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(2bfc3eaed583458b8381d4e7b4d36aef): 4925 bytes on disk
W20240502 08:22:10.338667 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:22:10.339315 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(2bfc3eaed583458b8381d4e7b4d36aef) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":659,"lbm_reads_lt_1ms":4}
W20240502 08:22:10.339802 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:10.340939 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6): 4586 bytes on disk
I20240502 08:22:10.341912 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 08:22:10.346628 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:10.352742 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:10.354933 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:10.406772 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:10.425948 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:10.437531 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:10.454452 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:10.468183 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:10.479207 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:10.494330 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:10.496716 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:10.542685 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:10.572254 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:10.581863 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:10.630983 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:10.668179 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(2bfc3eaed583458b8381d4e7b4d36aef) complete. Timing: real 0.683s user 0.044s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1079,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1508,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5045,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191207,"threads_started":2,"wal-append.queue_time_us":82797}
I20240502 08:22:10.675338 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(2bfc3eaed583458b8381d4e7b4d36aef): 4925 bytes on disk
I20240502 08:22:10.688566 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(2bfc3eaed583458b8381d4e7b4d36aef) complete. Timing: real 0.008s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7842,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240502 08:22:10.695745 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6): perf score=0.033574
W20240502 08:22:10.750831 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:10.685793 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.616s user 0.043s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10796,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":7857,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21795,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":61194,"threads_started":2,"wal-append.queue_time_us":50112}
I20240502 08:22:10.776222 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419): 4840 bytes on disk
I20240502 08:22:10.776891 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 08:22:10.854784 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:10.854903 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:10.957247 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.261s user 0.032s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14924,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":4387,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":832,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15111,"threads_started":2,"wal-append.queue_time_us":373}
I20240502 08:22:10.958753 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6): 4586 bytes on disk
I20240502 08:22:10.959447 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 08:22:11.017472 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.021396 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:11.044484 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:11.053589 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:11.076295 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.080582 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:11.081035 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:11.089633 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:11.092042 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:11.112557 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:11.141207 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:11.142025 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:11.144471 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:11.159332 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:11.162303 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:11.167621 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.213730 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c): perf score=0.033343
I20240502 08:22:11.231523 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:11.232239 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:11.280126 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(7b3f6a0b2561419ca6bd832fd65e3196): perf score=0.033360
W20240502 08:22:11.283061 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:11.297546 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.346796 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c): perf score=0.033368
I20240502 08:22:11.446031 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(7b3f6a0b2561419ca6bd832fd65e3196) complete. Timing: real 0.166s user 0.033s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10934,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":6240,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14711,"threads_started":2,"wal-append.queue_time_us":368}
I20240502 08:22:11.447535 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196): 4925 bytes on disk
I20240502 08:22:11.448206 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 08:22:11.454387 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.462419 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(7b3f6a0b2561419ca6bd832fd65e3196): perf score=0.033401
I20240502 08:22:11.481842 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.135s user 0.052s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10902,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2617,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11229,"threads_started":2,"wal-append.queue_time_us":394}
I20240502 08:22:11.483145 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(203e7914da3f4494b239c6603513caee): perf score=0.033400
W20240502 08:22:11.558385 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:11.560254 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.582039 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.368s user 0.057s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9037,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":7178,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32172,"threads_started":2,"wal-append.queue_time_us":22898}
I20240502 08:22:11.587157 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c): 4586 bytes on disk
I20240502 08:22:11.587879 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
W20240502 08:22:11.589064 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:11.594158 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:11.609390 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.126s user 0.047s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1503,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6467,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13648,"threads_started":1,"wal-append.queue_time_us":14110}
I20240502 08:22:11.613757 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee): 4840 bytes on disk
I20240502 08:22:11.614225 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(7b3f6a0b2561419ca6bd832fd65e3196) complete. Timing: real 0.151s user 0.061s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":24276,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":3677,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2929,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":50048,"thread_start_us":48776,"threads_started":2,"wal-append.queue_time_us":24624}
I20240502 08:22:11.615659 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196): 4925 bytes on disk
I20240502 08:22:11.616374 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240502 08:22:11.617547 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 08:22:11.619104 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c): 4586 bytes on disk
I20240502 08:22:11.620322 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240502 08:22:11.654070 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:11.682647 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:11.782507 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:11.794941 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:11.795560 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:11.845815 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:11.866263 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:11.881119 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:11.894151 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:12.053321 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:12.062701 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:12.070884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:12.075553 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:12.080451 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:12.085512 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:12.087946 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:12.096262 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:12.111399 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:12.137908 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.138401 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:12.279206 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:22:12.382422 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.394676 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.411151 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:12.442409 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:22:12.449898 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:12.458691 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling FlushMRSOp(8a45b1da348e4bb89f021df103a0603f): perf score=0.033348
I20240502 08:22:12.471388 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:12.476716 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:12.486848 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.541208 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:12.591500 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.596117 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(8a45b1da348e4bb89f021df103a0603f): perf score=0.033396
W20240502 08:22:12.604621 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:12.611279 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.620887 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: FlushMRSOp(8a45b1da348e4bb89f021df103a0603f) complete. Timing: real 0.162s user 0.074s sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":100349,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":5285,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":92348,"threads_started":2,"wal-append.queue_time_us":6356}
I20240502 08:22:12.621728 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542): perf score=0.033390
I20240502 08:22:12.622251 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f): 4925 bytes on disk
I20240502 08:22:12.622319 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.622872 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:22:12.630991 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling FlushMRSOp(12c17796e5d04014aafdac46984f4ba9): perf score=0.033397
W20240502 08:22:12.702863 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.730404 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:12.733278 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.739755 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:12.741640 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:12.799758 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.812173 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:12.826356 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:12.858570 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.875856 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.254s user 0.060s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10266,"dirs.run_cpu_time_us":292,"dirs.run_wall_time_us":5978,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3019,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":84612,"threads_started":2,"wal-append.queue_time_us":54998}
I20240502 08:22:12.877292 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542): 4586 bytes on disk
I20240502 08:22:12.877964 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 08:22:12.953997 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:12.968905 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:12.982342 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:12.990491 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:13.005937 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.013450 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:13.016242 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:13.018780 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:13.031208 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:13.066947 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(8a45b1da348e4bb89f021df103a0603f) complete. Timing: real 0.470s user 0.050s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51423,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":7564,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1057,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":73060,"threads_started":2,"wal-append.queue_time_us":24497}
I20240502 08:22:13.068802 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542): perf score=0.033522
W20240502 08:22:13.107223 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:13.156265 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.175218 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: FlushMRSOp(12c17796e5d04014aafdac46984f4ba9) complete. Timing: real 0.544s user 0.037s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11528,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":4104,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12833,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":82942,"threads_started":2,"wal-append.queue_time_us":57512}
I20240502 08:22:13.177206 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9): 4840 bytes on disk
I20240502 08:22:13.179090 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:22:13.186787 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.116s user 0.033s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":925,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1126,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27978,"threads_started":1,"wal-append.queue_time_us":27785}
I20240502 08:22:13.188403 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f): 4925 bytes on disk
I20240502 08:22:13.189093 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240502 08:22:13.191164 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542): 4586 bytes on disk
I20240502 08:22:13.191848 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
W20240502 08:22:13.290787 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.339906 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:13.382761 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:13.421152 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:13.463835 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:13.483017 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:13.483897 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:13.484381 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:13.531071 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:13.566171 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:22:13.611212 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:13.647264 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:13.706578 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:13.721539 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:13.746304 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.753317 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:13.779510 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.798319 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:13.804983 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 96 similar messages]
I20240502 08:22:13.807478 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:13.827217 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:13.858127 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 68549.9us
W20240502 08:22:13.862275 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.960088 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:13.960987 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:13.984704 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:13.989217 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:13.991138 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:14.003394 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:14.024798 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:14.040668 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:14.066998 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:14.074211 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:14.084421 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:14.220700 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:14.234324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:14.257001 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:14.285533 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:14.314325 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:14.318341 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:14.331363 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:22:14.347597 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:14.406244 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:14.471798 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:14.526975 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:14.542073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:14.548118 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:22:14.548941 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:14.550805 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:14.553644 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:14.579237 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:14.601644 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:14.668697 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:14.696995 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:14.721911 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:14.788532 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:22:14.892967 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:14.917516 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:14.918047 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:14.920843 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:14.932416 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:14.935226 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:14.944511 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:14.951619 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:14.999919 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:15.005803 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:15.029834 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.161070 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 157060us
W20240502 08:22:15.174626 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:15.174968 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:22:15.317265 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.324836 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.367017 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:22:15.407756 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.424005 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:15.455574 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.529191 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:22:15.533852 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:15.567826 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:15.599498 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:15.620282 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:15.634159 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:15.647342 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:15.675844 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:15.696758 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:15.731657 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:15.739320 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:15.747480 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:15.802181 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:15.820148 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:15.831975 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:15.887354 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:15.899995 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:15.903980 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:15.905390 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:15.918236 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:15.946523 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:15.957125 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:15.971021 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:15.992770 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:16.008842 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:16.032356 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:16.136758 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:16.269732 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:16.291833 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:16.306032 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:16.310866 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:16.327148 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:16.334304 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:16.359786 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:16.383100 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.400272 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.431807 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.443146 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.447412 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.520503 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:16.556458 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:16.668900 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:16.686230 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:16.689927 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:16.732023 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:16.736979 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:16.754283 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:16.765435 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:16.939625 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:17.011348 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.018059 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:17.026042 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:17.057384 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:17.060036 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:17.063642 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:17.104602 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:17.105137 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 44564.5us
I20240502 08:22:17.173688 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:17.202723 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:17.208639 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:17.209461 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:17.214149 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:17.266494 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:22:17.300105 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.316500 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:17.321298 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:17.341696 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:17.342985 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.350545 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:17.410140 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:17.411212 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:17.541172 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:17.582777 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:17.606757 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:17.621111 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:22:17.669057 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:17.725134 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:17.730458 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:17.732070 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.748507 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:17.763307 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:17.782407 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:17.826797 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:17.931988 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:17.932910 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.949986 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:17.963236 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:17.971385 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:17.972697 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:17.976924 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:17.988184 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:18.039068 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:18.045732 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:18.046334 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:18.075079 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:18.086462 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:18.105736 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:18.148460 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:18.172362 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:18.208323 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:18.208778 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:18.215833 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:18.248695 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:18.372313 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:18.394824 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:18.395910 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:18.396293 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:18.403493 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:18.409368 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:18.543768 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:18.592679 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 08:22:18.593997 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:18.610534 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:18.680209 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:18.740851 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:18.749660 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:18.757151 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:18.759217 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:18.807310 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 08:22:18.830593 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:18.840235 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:18.861061 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:18.917038 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:19.031507 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:19.088981 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:19.092694 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:19.195940 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:19.196538 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:19.200317 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:19.209901 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.224718 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:19.264743 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:19.289294 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.293632 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:19.300195 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:19.330905 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:19.345988 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:19.346549 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:19.367238 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:19.388530 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:19.402863 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:19.444175 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:19.488806 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:19.494735 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:19.528465 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:19.587435 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:19.597210 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:19.622233 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:22:19.629369 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:19.642527 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:19.655432 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.671940 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:19.681684 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:19.765564 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:19.773964 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.777179 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.801769 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:19.835752 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:19.845559 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:19.859498 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:19.873811 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:19.893802 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:19.942150 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:19.970893 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:20.019135 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:22:20.022567 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:20.030009 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:22:20.153676 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.190402 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.252259 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.288664 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:20.294761 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:20.314611 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.322727 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:22:20.361405 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:20.370618 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:20.402913 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:22:20.404774 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:20.425573 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:20.477411 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:20.481582 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:20.493497 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:20.565968 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:20.566843 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.628954 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.713475 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:20.745555 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:22:20.763576 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:20.817603 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:20.821019 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:20.832201 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:20.832480 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:20.833187 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:20.834215 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:20.836990 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:20.863296 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:20.880760 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:20.888425 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:20.922149 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:20.926898 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:20.944692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:20.961584 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:21.017966 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:21.024108 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.039227 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:21.047569 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:21.060194 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:21.119333 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.136552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:21.137171 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:21.330005 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:21.360680 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:21.370716 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.384189 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:21.385445 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:21.386745 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.411895 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:21.412901 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:21.464267 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:21.469195 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.489691 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:21.512840 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:21.521485 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:21.565676 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:21.623812 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:21.736864 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:21.799163 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:21.902246 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:21.942838 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:21.966341 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:21.980291 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:21.996368 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:21.997429 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:22.000746 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:22.022673 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:22.037180 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:22.044123 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:22.078472 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:22.098843 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:22.125712 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:22.148401 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:22.220103 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:22.273106 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:22.302825 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:22.364642 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:22.397369 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:22:22.403998 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:22.430859 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:22.455277 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:22.463503 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:22.478683 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:22.491565 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:22.496953 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:22.519573 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:22.546984 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:22.554814 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:22.581884 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:22.598937 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:22.603224 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:22.607611 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:22.617671 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:22.652904 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:22.682807 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:22.728540 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:22.758309 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:22.776680 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:22.807101 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:22.834249 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:22.886070 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:22.913864 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:22.914597 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:22.947080 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:22.950384 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.009734 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.039826 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.071738 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:22:23.194770 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:23.231102 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:23.265893 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:23.267035 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:23.290393 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:23.296708 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:23.301750 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:23.365039 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:23.394469 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:23.488663 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:23.489182 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:23.510778 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:23.513144 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:23.541613 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:23.566618 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:23.578624 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.648299 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.666626 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 08:22:23.667755 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:23.703303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:23.737598 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.743453 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:23.745775 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:23.760162 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:23.764586 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:23.791718 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:23.811467 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:23.814728 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:23.837384 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:23.856879 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:23.908274 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:23.928162 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 08:22:23.938238 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:23.983830 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:24.009415 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:24.018679 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:24.024264 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:24.045260 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:24.047787 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:24.090654 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:24.119712 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2): perf score=0.033400
I20240502 08:22:24.147722 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:24.213598 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:24.236564 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:24.284967 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:24.310137 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:24.355414 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:24.360589 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:24.380303 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:24.416545 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 63914.4us
W20240502 08:22:24.419006 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:24.445611 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:24.471865 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:24.616624 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:24.644513 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:22:24.693172 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.573s user 0.141s sys 0.107s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":27357,"dirs.queue_time_us":53446,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":7977,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":170788,"threads_started":3,"wal-append.queue_time_us":90281}
I20240502 08:22:24.695555 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2): 4838 bytes on disk
I20240502 08:22:24.696385 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(3898bfdb9c3d4a05af78f79fd2de61e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 08:22:24.724107 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:24.832007 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:24.970965 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:24.976666 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:24.993141 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:24.996538 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.005210 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:25.009749 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.029771 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:25.037671 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:25.115782 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:25.129056 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:25.132441 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:22:25.150022 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:25.155107 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.159317 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:25.173869 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 46202us
W20240502 08:22:25.182888 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.191562 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:25.191941 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.193313 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:25.213146 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:25.275126 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:22:25.291183 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:25.316553 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:25.323621 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.324340 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.334988 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:25.380092 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:25.391446 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.407655 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:25.419514 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:25.424141 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 08:22:25.425545 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.431309 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:25.493853 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.496333 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:25.498741 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.518855 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:25.524911 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:22:25.553226 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:25.559893 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:25.563467 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:25.571331 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:25.640002 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:25.720988 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:25.792377 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:25.827497 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:26.005976 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:26.034371 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:22:26.134343 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:26.191668 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.193528 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:26.253270 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:26.255884 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.268419 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:26.269057 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:26.278270 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.319077 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:26.332682 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:26.336798 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 50323us
I20240502 08:22:26.351508 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:26.375898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:26.404667 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:26.405575 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.455987 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(76bb1e3e2b9d4e31bf93ac1ce08a7833): perf score=0.033380
W20240502 08:22:26.478211 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:26.548207 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.583326 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(76bb1e3e2b9d4e31bf93ac1ce08a7833) complete. Timing: real 0.127s user 0.063s sys 0.033s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8012,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":19579,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1042,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":45884,"threads_started":2,"wal-append.queue_time_us":37697}
I20240502 08:22:26.584967 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833): 4840 bytes on disk
I20240502 08:22:26.585821 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4}
I20240502 08:22:26.587082 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419): perf score=0.033404
I20240502 08:22:26.600567 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:26.634788 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:26.666791 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:26.682538 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.095s user 0.078s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":5039,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1126,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6521,"threads_started":1,"wal-append.queue_time_us":6402}
W20240502 08:22:26.683804 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:26.684554 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419): 4840 bytes on disk
I20240502 08:22:26.685133 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(efec2cc693bd4dd992d0bec22fca5419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:22:26.696666 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:26.758708 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:26.825928 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:26.868254 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:26.885190 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c): perf score=0.033393
I20240502 08:22:26.969691 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:26.996368 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:27.009997 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:27.023789 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:27.073755 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:27.099730 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:27.104040 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:27.110860 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:27.117055 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:27.137842 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:27.165439 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:27.172904 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:27.179348 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:27.207202 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.252712 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:27.260075 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.332960 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:27.338585 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:27.359385 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:22:27.360431 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:27.380815 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:27.384070 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:27.403774 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:27.406118 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.520s user 0.060s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":357,"dirs.queue_time_us":231301,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":3794,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":882,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":288492,"threads_started":3,"wal-append.queue_time_us":54538}
I20240502 08:22:27.407807 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(1648586b889b4a1e9ac34481cec6257a): perf score=0.033521
I20240502 08:22:27.419695 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:27.433439 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:27.472355 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:27.511878 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:27.629422 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.633793 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.194s user 0.073s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":3680,"drs_written":1,"lbm_read_time_us":182,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1129,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19681,"threads_started":1,"wal-append.queue_time_us":19785}
I20240502 08:22:27.635615 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(cec56facd7f740aeb42bc7c32495dae3): perf score=0.033461
W20240502 08:22:27.682822 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.692425 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68): perf score=0.033358
W20240502 08:22:27.696924 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:27.697386 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:27.704886 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:27.769413 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 84679.7us
W20240502 08:22:27.864801 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.877633 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.242s user 0.097s sys 0.100s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1732,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":183262,"threads_started":1,"wal-append.queue_time_us":183340}
I20240502 08:22:27.878748 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3): 4840 bytes on disk
I20240502 08:22:27.879750 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 08:22:27.880738 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a): 4586 bytes on disk
I20240502 08:22:27.881491 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(1648586b889b4a1e9ac34481cec6257a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:22:27.882285 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c): 4586 bytes on disk
I20240502 08:22:27.882746 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(d62f6eec9ee04f93a3738a83f790292c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 08:22:27.912766 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.220s user 0.060s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9058,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":7347,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":951,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":100580,"threads_started":2,"wal-append.queue_time_us":91419}
I20240502 08:22:27.914199 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68): 4925 bytes on disk
I20240502 08:22:27.923997 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(599d5781f1e24ae9ad425a34d88c6b68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240502 08:22:27.983736 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:27.991066 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:27.991060 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.021507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:28.048748 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:28.055835 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:28.069525 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:28.079226 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:28.089510 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:28.092262 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:28.143632 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:28.157436 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:22:28.189085 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:28.243716 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.253443 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:28.262566 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:28.276129 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:28.291436 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:28.294708 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:28.344980 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.428334 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(07f5c2dfcd974ee2b8720e9eb052c3f7): perf score=0.033368
W20240502 08:22:28.550191 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:28.609196 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.616791 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:28.631052 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:28.631456 21924 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:28.646440 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:28.650990 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.667178 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:28.681479 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 08:22:28.690110 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:28.702023 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:28.702782 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:28.744709 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(07f5c2dfcd974ee2b8720e9eb052c3f7) complete. Timing: real 0.316s user 0.067s sys 0.057s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13233,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":21451,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":146424,"threads_started":2,"wal-append.queue_time_us":133469}
I20240502 08:22:28.746125 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7): 4925 bytes on disk
I20240502 08:22:28.752436 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 08:22:28.757540 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:28.758201 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.777362 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:28.784168 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:28.824132 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:28.828200 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:28.829396 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:28.830847 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:28.856329 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:29.000046 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240502 08:22:29.013417 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): perf score=0.033356
W20240502 08:22:29.020589 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:29.071746 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:29.084372 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:29.105399 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:29.113978 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:29.118536 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:29.146850 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:29.165199 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:29.203331 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:29.238246 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:29.311777 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:29.321780 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.308s user 0.055s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6432,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":2050,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":909,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160829,"threads_started":2,"wal-append.queue_time_us":134229}
I20240502 08:22:29.323272 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): 4924 bytes on disk
I20240502 08:22:29.323976 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240502 08:22:29.324105 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:29.405234 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:29.422116 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:29.465687 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:29.516445 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:29.543355 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:29.560706 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:29.650496 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:29.663467 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:22:29.695988 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:22:29.731930 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:29.739470 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:29.744861 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:29.752152 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:29.764282 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:29.769065 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:29.775656 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:29.789774 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:29.834831 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:29.945708 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:29.946067 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:29.979063 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:30.003077 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:30.029115 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:30.041589 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:30.041827 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:30.066408 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:30.069166 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:30.107813 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:30.123759 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:30.132870 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:30.154727 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:22:30.162900 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:30.179433 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:30.209486 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:30.210187 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:30.211144 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:30.223942 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.286926 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.303848 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:30.309024 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:30.331461 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542): perf score=0.033391
I20240502 08:22:30.393918 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(a20b2fd9f1ce488ba7793ed7ac268c16): perf score=0.033352
I20240502 08:22:30.460274 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.128s user 0.038s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5042,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":8954,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32234,"threads_started":2,"wal-append.queue_time_us":27158}
I20240502 08:22:30.461727 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542): 4586 bytes on disk
I20240502 08:22:30.463745 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(d8042f6ab6404c5a9f8cce0b8f971542) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 08:22:30.469029 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.472074 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.473184 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.532207 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:22:30.533610 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:30.632261 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(a20b2fd9f1ce488ba7793ed7ac268c16) complete. Timing: real 0.238s user 0.078s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":570,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1678,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1071,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6755,"threads_started":2,"wal-append.queue_time_us":6350}
I20240502 08:22:30.633780 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(a20b2fd9f1ce488ba7793ed7ac268c16): 4840 bytes on disk
I20240502 08:22:30.634419 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(a20b2fd9f1ce488ba7793ed7ac268c16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 08:22:30.653800 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:30.663959 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:30.684732 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:30.766497 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:30.788290 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:30.821892 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:30.822006 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 144598us
W20240502 08:22:30.842883 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:30.859972 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:30.886648 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): perf score=0.033388
I20240502 08:22:30.923238 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:30.938915 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:30.954558 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:22:31.031932 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:31.036049 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:31.078871 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:31.098583 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:31.101477 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:31.108696 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:31.116297 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:31.128424 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:31.134119 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:31.134157 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:31.149566 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:31.155755 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:31.162333 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:31.162930 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:31.182317 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:31.198062 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:31.207706 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:31.208160 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:31.208717 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:31.251808 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:31.291872 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:31.359664 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.473s user 0.056s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":149581,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":7988,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":162294,"threads_started":2,"wal-append.queue_time_us":8595}
I20240502 08:22:31.361369 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa): 4924 bytes on disk
I20240502 08:22:31.362212 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(6f04a3972e384fa3b0ffe0cb8a4f35aa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4}
W20240502 08:22:31.406569 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:31.415830 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:31.430770 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:31.468693 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:31.469604 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:31.505148 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:31.521730 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:31.552089 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:31.574239 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:22:31.620373 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:31.680951 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:31.702744 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:31.721349 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(12c17796e5d04014aafdac46984f4ba9): perf score=0.033344
I20240502 08:22:31.731814 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:31.733294 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:31.775802 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:31.777199 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:31.821393 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:31.893417 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:31.927551 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(12c17796e5d04014aafdac46984f4ba9) complete. Timing: real 0.206s user 0.044s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26034,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":2844,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":838,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39347,"threads_started":2,"wal-append.queue_time_us":7184}
I20240502 08:22:31.928941 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9): 4840 bytes on disk
I20240502 08:22:31.929608 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 08:22:31.930794 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(203e7914da3f4494b239c6603513caee): perf score=0.033400
I20240502 08:22:32.017387 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.086s user 0.049s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":8781,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1625,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":37960,"threads_started":1,"wal-append.queue_time_us":35782}
I20240502 08:22:32.020558 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee): 4840 bytes on disk
I20240502 08:22:32.021757 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240502 08:22:32.025962 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:32.106791 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:32.113566 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:32.116899 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:32.116847 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6): perf score=0.033346
I20240502 08:22:32.136873 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:32.148183 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:32.150219 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:32.159714 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:32.164680 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:32.193925 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:32.208606 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:32.352663 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.235s user 0.040s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":19215,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":5386,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3991,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32502,"threads_started":2,"wal-append.queue_time_us":6841}
I20240502 08:22:32.355755 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6): 4586 bytes on disk
I20240502 08:22:32.356376 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240502 08:22:32.397392 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 59727.4us
I20240502 08:22:32.446161 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:32.459662 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:32.469399 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:32.475330 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:32.475749 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:32.496773 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:32.528821 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:32.539093 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:32.552642 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:32.553421 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:32.554895 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:32.580649 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:32.594806 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:32.603119 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:22:32.604076 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:32.604476 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:32.608903 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(2de41d6c6de0400397645c05a0fa17de): perf score=0.033356
I20240502 08:22:32.668388 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:32.701174 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:32.704197 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:32.809028 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:32.853029 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:32.893640 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.284s user 0.052s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11193,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":2007,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":30066,"threads_started":2,"wal-append.queue_time_us":4256}
I20240502 08:22:32.895212 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de): 4925 bytes on disk
I20240502 08:22:32.895864 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240502 08:22:33.012509 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.021440 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:33.034634 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:33.035075 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.103927 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.197801 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:33.232198 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:33.242365 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:33.273495 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:33.279541 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:33.289485 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:33.292876 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.385201 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.399017 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(dc49c050280346db88d1323e30513c80): perf score=0.033373
I20240502 08:22:33.405643 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:33.413868 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:33.422565 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:33.435798 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:33.445050 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:33.451282 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.453467 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:33.465543 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.500473 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:33.512394 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:33.514927 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:33.532230 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:33.544381 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:33.547494 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(59f5d97e6e0a4755be41699246907e55): perf score=0.033385
W20240502 08:22:33.572088 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.573724 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:33.586890 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:33.595458 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:33.607450 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.629961 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:33.650979 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.671509 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:33.678824 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:33.689079 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:33.691401 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:22:33.692970 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.696408 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 08:22:33.712810 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(dc49c050280346db88d1323e30513c80) complete. Timing: real 0.314s user 0.035s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22135,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":4180,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":46666,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":46200,"threads_started":2,"wal-append.queue_time_us":2016}
I20240502 08:22:33.714704 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(dc49c050280346db88d1323e30513c80): 4586 bytes on disk
I20240502 08:22:33.717972 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:33.726370 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:33.732242 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(dc49c050280346db88d1323e30513c80) complete. Timing: real 0.006s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":5064,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
W20240502 08:22:33.764812 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.780336 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:33.783571 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:33.862784 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:33.914517 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(59f5d97e6e0a4755be41699246907e55) complete. Timing: real 0.366s user 0.049s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":403,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1758,"drs_written":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":42729,"threads_started":2,"wal-append.queue_time_us":9363}
I20240502 08:22:33.945955 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(80b3ba1d834a4314b63f336675845441): perf score=0.033445
W20240502 08:22:34.003824 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240502 08:22:34.020625 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.074s user 0.044s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":219,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1869,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22060,"threads_started":1,"wal-append.queue_time_us":22183}
I20240502 08:22:34.022176 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(59f5d97e6e0a4755be41699246907e55): 4840 bytes on disk
I20240502 08:22:34.024099 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(59f5d97e6e0a4755be41699246907e55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:22:34.025481 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441): 4587 bytes on disk
I20240502 08:22:34.026103 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(80b3ba1d834a4314b63f336675845441) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 08:22:34.094967 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:34.115010 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:34.202577 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:34.205067 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:34.218842 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:34.238873 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:34.259521 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:34.260644 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:34.407033 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:34.447815 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:34.461392 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:34.481184 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:34.486821 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:34.524288 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:34.551779 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:34.641028 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:34.778239 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:22:34.786981 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:34.794101 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:34.817209 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:34.855649 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 56652.1us
W20240502 08:22:34.864252 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:34.913635 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:34.921561 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:34.971155 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:34.979425 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:34.998224 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:35.014539 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.038293 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:35.040421 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:35.041124 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.116072 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:35.135159 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:35.159718 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:35.184245 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:35.200819 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:22:35.206192 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:35.217751 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:35.266487 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.266882 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:35.287878 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:35.294065 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 55413.4us
W20240502 08:22:35.296641 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.297919 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:35.351330 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:35.351980 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:35.352051 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:35.381470 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:35.409229 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:35.504871 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:35.515978 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:35.607527 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:35.698789 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:35.720451 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:22:35.722434 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.757115 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:35.774145 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:35.798635 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.817961 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:35.819142 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:35.864903 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:35.878760 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:35.911257 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:35.925123 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:35.937201 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:22:35.961459 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:36.012025 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:36.033552 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:36.073328 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.112804 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:36.120003 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.130312 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:36.132009 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.138082 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:36.150653 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:36.152557 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:36.198010 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:36.226999 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:36.234158 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:36.252504 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:22:36.285738 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:36.287598 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:36.296864 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:36.316109 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.316644 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:36.338131 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:36.378872 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:36.438344 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:36.481911 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:36.609705 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.670382 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:36.709096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:36.741977 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:36.745749 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:22:36.772408 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:36.828867 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:36.844991 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:36.849831 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:36.854007 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:22:36.953258 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:36.965513 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:36.988610 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:36.993729 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:37.000628 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:37.015419 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:37.094820 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:37.166193 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:37.177667 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:37.201552 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:37.227795 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:37.228201 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:37.253991 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:37.262542 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:37.327344 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:37.344098 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:37.344568 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:37.375411 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:37.404033 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:37.436785 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:37.469713 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:37.503276 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:37.509680 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:37.578085 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:37.642704 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:37.677206 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:37.712277 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:37.769152 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:37.788240 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:37.788636 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:37.810845 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:37.816946 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:37.841070 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:37.847059 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:37.852317 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:37.862064 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:37.944419 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:37.985605 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.101737 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:38.127318 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.143195 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.175207 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.216154 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.252916 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:38.253954 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:38.281878 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:38.301723 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:38.344457 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.347215 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:38.417218 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.417842 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.431306 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:38.462078 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:38.470723 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:38.477890 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:38.483886 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.484797 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.493047 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:38.494871 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:38.501088 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:38.511914 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.523437 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:38.571000 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:38.580832 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:38.592638 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:38.616878 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:38.620003 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.620710 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.626502 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.640569 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.671506 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:38.671981 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:38.682289 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:38.694911 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:38.716049 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240502 08:22:38.721699 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:38.739485 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:38.760264 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.776357 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.857087 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:22:38.862124 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:38.964946 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:38.997058 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:39.005648 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:22:39.126127 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:39.128122 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:39.135174 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:39.173465 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:39.175655 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:39.179665 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:39.294657 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:39.307819 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:39.327147 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:39.447844 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:39.607019 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:39.630704 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:39.658562 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:39.670915 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:39.700224 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:39.713683 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:39.719484 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:39.760040 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:39.766700 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:39.792660 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:22:39.795336 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:39.812230 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:39.841497 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:39.852918 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:39.857556 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:39.859086 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:39.867370 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:39.907213 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:40.023178 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.084895 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:40.101225 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:40.101401 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:40.134145 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:40.136027 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:40.137616 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:40.139154 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:40.165935 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:40.173024 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:40.177744 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:40.178143 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:40.183789 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:40.206552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:40.217123 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:40.241228 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.263033 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.324035 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:40.339568 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.341735 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:40.362617 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.362778 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:40.502465 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:40.509686 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 08:22:40.559129 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:40.574390 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:40.579756 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:40.590507 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:40.609946 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:40.621140 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.764322 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:22:40.777086 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:22:40.778632 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:40.803812 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:40.874370 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.928056 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:40.949630 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:40.981892 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:40.989733 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:41.016474 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:41.018342 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:41.041229 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:41.047045 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:41.078471 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:41.085672 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:41.143235 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:41.184842 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:41.195731 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:41.204129 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:41.206418 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:41.208109 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:41.241421 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:41.255640 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:41.279080 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:41.292047 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:41.321110 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:41.341364 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:41.349658 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:41.370249 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:41.406989 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:41.429560 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:22:41.445652 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:41.447754 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:41.463647 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:41.491565 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:41.501175 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:22:41.504333 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:41.607936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:41.628304 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:41.635843 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:41.636782 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:41.660189 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:41.671644 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:41.749655 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:41.751606 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:41.824312 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:41.834932 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:41.967722 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:42.010865 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:42.015925 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:42.041482 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:42.060412 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:42.072604 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:42.098963 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:42.241294 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:42.241798 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.297281 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:42.310631 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:42.323817 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:22:42.329872 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:42.362975 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:42.372010 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.381784 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:42.421741 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:42.432758 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:42.488158 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:42.509778 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:42.531354 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:42.552071 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:42.553212 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:42.584913 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:42.600567 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:42.601233 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.693241 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.722985 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:42.767668 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.786513 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.841893 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:42.850885 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:42.921205 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:43.051884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:43.078213 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:43.090620 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:43.094424 21485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 70246us
I20240502 08:22:43.109997 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:43.139632 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:43.142597 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:43.143487 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:43.175873 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:43.182324 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:43.184623 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:43.213532 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:43.238320 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:43.265534 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:43.291647 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.330533 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.340468 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:43.401542 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:43.416783 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 57033.6us
W20240502 08:22:43.428184 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.447767 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:43.465829 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.482152 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:43.519462 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:43.523928 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:43.524186 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:43.596841 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:43.611104 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:43.642679 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:43.739935 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.787343 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.797827 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:43.883574 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:22:43.955634 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:43.956182 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:43.956763 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:43.964102 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:43.980931 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:43.994377 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:43.994887 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:44.020125 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:44.031761 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:44.048182 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:22:44.050133 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:44.069315 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:44.087814 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:44.116289 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:44.117156 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:44.156315 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:44.224378 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:22:44.231799 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:44.241541 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:44.335979 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:44.379154 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:44.435737 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:44.443137 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:44.460346 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 65099us
I20240502 08:22:44.463399 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:44.477316 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:44.486886 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:44.517156 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:44.522743 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:44.564131 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:44.569295 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:44.627957 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:44.629415 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:44.798877 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:44.809549 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:44.819811 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:44.823019 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:44.840121 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:44.899820 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:44.938282 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 71194.3us
I20240502 08:22:44.967310 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:44.969201 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:44.987681 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:44.992115 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:22:44.993605 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:44.993870 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:45.008818 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:45.013077 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:45.024660 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:45.081943 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:45.118367 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.118584 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.128453 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:22:45.179975 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.232383 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.232682 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:45.333317 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:45.341171 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:45.380259 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:45.416332 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 64774.4us
I20240502 08:22:45.418151 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:45.419449 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:45.445598 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:45.450807 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:45.465181 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:45.544775 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.552966 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:22:45.690521 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:45.728044 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:45.739670 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:45.810364 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:45.836566 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:45.866400 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:45.872557 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:45.875622 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:45.897388 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:45.929833 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:22:45.934607 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:45.947283 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:45.957957 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:45.958817 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:45.971165 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:46.010882 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:46.018200 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:46.072249 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:46.110139 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:46.143218 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:46.145515 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:46.162158 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:46.176894 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:46.185290 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:46.325572 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:46.361161 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:46.378732 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:46.399315 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:46.400965 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:46.401782 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:46.446449 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:46.457067 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:46.469674 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:46.569476 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:46.590935 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:46.657949 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:46.708045 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:46.760412 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:22:46.768733 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:46.806807 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:46.807788 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:46.821657 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:46.833591 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:46.842772 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:46.858412 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:46.887892 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:46.926605 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:46.933735 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:46.934736 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:46.954772 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:46.979285 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:46.993913 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:47.005184 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:47.021003 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:47.027405 23851 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:47.027734 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:47.052017 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:47.064239 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:47.066895 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:47.067617 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:47.118265 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:47.204353 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:47.252683 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:47.288694 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:47.295432 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:47.296772 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:47.318773 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:47.328338 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:47.388075 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:47.394644 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:47.407121 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:47.444581 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:47.485270 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:22:47.538116 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:47.552946 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:47.569129 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:47.583344 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:47.599679 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:47.612064 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:47.630899 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:47.660089 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:47.700870 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:47.710239 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:47.730540 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:47.847074 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:47.903088 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:22:47.914423 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:47.953714 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:47.997493 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:48.020511 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 55202.6us
I20240502 08:22:48.042408 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:48.088904 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:48.120707 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:48.141155 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:48.143858 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:48.145030 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.167151 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:48.167939 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:48.177449 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:48.183025 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.184001 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:48.198055 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:48.267225 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.273342 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:48.328660 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:48.387169 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:48.428664 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.531889 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:48.548533 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:48.582422 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:48.600834 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.620849 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:48.631924 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:48.652304 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:48.729648 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:48.764492 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:48.769001 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:48.797029 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:48.831642 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:48.853578 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:48.903144 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:48.904387 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:48.919868 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:48.928674 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:48.938436 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:48.977862 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:49.060354 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:22:49.067235 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:22:49.071334 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:49.105412 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:49.118850 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:49.136691 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:49.156056 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:49.170733 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:49.193670 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:49.195765 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:49.221580 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:49.274722 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:49.366356 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:49.389146 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:49.389417 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:22:49.474155 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:49.479502 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:49.485215 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:49.492769 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:49.504547 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:49.513283 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:49.563953 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:49.566316 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:49.617446 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:49.634963 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:49.645926 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:49.676069 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:49.691197 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 74993.2us
I20240502 08:22:49.711658 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:49.750411 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:49.754119 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:49.792187 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:49.833923 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:49.834846 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:49.899152 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:49.944383 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:49.999012 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:50.024713 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:22:50.048298 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:50.082911 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:50.145920 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:50.146672 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:50.147045 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:50.148540 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:50.191344 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:50.231156 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:50.233927 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:50.250208 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:50.251824 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:50.273788 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:50.418376 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:50.451830 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:50.453064 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:50.488683 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:50.497025 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:22:50.537484 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:50.561148 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:50.579536 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:22:50.609889 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:50.700640 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:50.716514 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:50.751515 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:50.772984 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:50.813503 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:50.818367 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:50.819959 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:50.843022 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:50.898867 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:50.912117 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:50.912292 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:50.937319 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:50.937842 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:50.947062 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:50.963261 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:50.970968 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:51.015704 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:51.037039 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:51.052928 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:22:51.107671 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:51.179903 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:51.184074 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.191161 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:51.205833 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:51.219748 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.233008 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:51.233819 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:51.277702 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:51.278997 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:51.304522 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:51.364184 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.517300 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.582594 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:51.613878 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:22:51.673794 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.691964 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.736006 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:51.767452 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:51.806293 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:51.817462 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:51.827491 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:51.861168 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:51.872121 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:51.884284 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:51.931098 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:51.958171 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:51.974062 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:51.995730 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:52.013809 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:52.015345 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:52.079900 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:52.094679 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:52.112393 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:52.127398 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:52.150712 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:52.176875 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:52.222015 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:52.232287 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:52.234200 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:52.235129 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:52.313024 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:52.349916 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:52.428164 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:52.443814 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:52.445379 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:52.453795 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:22:52.464169 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:52.500263 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:52.501180 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:52.520663 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:52.522114 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:52.523396 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:52.564079 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:52.598246 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:52.622386 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:52.656181 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:52.681730 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:52.682766 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:52.692385 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:52.693248 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:52.711566 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:52.771448 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:52.780874 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:52.872478 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:53.016225 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:53.050221 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.063822 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.089977 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:53.153399 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:53.179003 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.179876 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:53.194136 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:53.199425 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:53.288064 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:22:53.302834 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:53.330353 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.365505 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.439205 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:53.503024 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:53.528829 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:53.545737 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.554406 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.606232 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.618477 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:53.620468 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:53.624553 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.640045 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:53.667843 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:53.695986 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:53.705466 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:53.705904 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:53.730275 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:53.732961 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:53.737125 23406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 54716.3us
I20240502 08:22:53.739387 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:53.742585 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:53.772713 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.799660 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:53.806084 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:53.831337 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:53.870149 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:53.910148 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:53.912616 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:53.977624 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:53.985200 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:54.064733 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:54.076536 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:22:54.097010 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:54.112561 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 08:22:54.118428 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:54.141119 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:22:54.171727 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:22:54.182459 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:54.185124 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:54.188947 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:54.265275 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:54.332458 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:54.374147 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:54.386567 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:54.427284 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:54.569698 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:54.653471 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:54.703666 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:54.779733 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:54.796093 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:54.815407 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:54.830739 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:54.832633 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:54.833362 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:54.860100 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:54.864490 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:54.912366 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:54.974898 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:54.980615 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:54.980937 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:55.006877 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:55.022348 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:55.041908 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:55.071732 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:22:55.080601 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:55.085119 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:55.103016 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:55.110402 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:55.121340 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:55.126585 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:55.139185 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:55.148530 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:55.151669 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:55.191272 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:55.208213 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:55.217552 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:55.286691 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:55.323472 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:55.332120 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:55.341234 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:55.347303 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:55.348232 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:55.380829 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:55.402400 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:55.412389 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:55.451229 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:55.496579 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:55.628718 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:22:55.657730 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:55.663477 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:55.787552 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:55.810346 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:55.835515 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:55.841965 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:55.859580 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:55.887154 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:55.900636 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:55.920601 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:55.947199 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:55.955600 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:55.955778 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:55.957542 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:55.967314 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.023129 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.023854 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:56.027299 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:56.028062 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:56.031807 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.073588 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:56.097965 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.117296 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:56.126217 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:56.162734 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:56.248756 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.348994 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.349920 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.361472 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.425110 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:22:56.439092 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.448105 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:56.466521 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.480870 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:56.518146 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:56.519027 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:56.538009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:56.538818 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:56.544334 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:56.545904 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:56.573765 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:56.587771 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:56.603152 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:56.612938 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:56.642781 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:56.658855 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:56.723284 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:22:56.786266 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:56.807037 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:56.888190 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:56.949213 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:57.101253 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:57.175398 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:57.202373 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:57.206063 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:57.207403 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:57.260334 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:57.260674 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:57.268041 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:57.270897 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.288265 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:57.302023 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:57.305532 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:57.316187 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:57.328814 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:57.330040 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.344235 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:57.354657 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:57.376885 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:57.382943 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:22:57.397614 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:57.448307 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:57.501046 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.517567 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:57.521509 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:57.527283 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.551625 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:57.554232 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:57.557035 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:57.644948 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:57.657130 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:57.695483 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.697378 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:57.712711 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:57.713418 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:57.717633 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:57.722872 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:57.727125 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:57.728044 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:57.736075 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:58.074908 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:58.135304 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:58.143108 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.168671 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:58.181325 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:58.188066 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:58.209194 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:58.211269 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:58.243714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:58.248154 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:58.329995 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:58.462661 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:58.465204 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:22:58.470562 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.503351 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:58.530265 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:58.547765 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:22:58.549227 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:58.555020 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:58.572268 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:58.589753 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:58.597427 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:58.608590 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:58.631481 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.634191 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:58.635807 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:58.656921 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:58.668816 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:58.677332 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.702106 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:58.757932 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:58.836565 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:58.854068 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e): perf score=0.033359
W20240502 08:22:58.864176 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.866603 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:22:58.877985 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:22:58.909250 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:58.915565 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:58.921026 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.067s user 0.036s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8651,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":16712,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":915,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9407,"threads_started":2,"wal-append.queue_time_us":535}
I20240502 08:22:58.921286 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:58.922430 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e): 4840 bytes on disk
I20240502 08:22:58.923132 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 08:22:58.924918 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:58.965318 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:59.005672 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:59.018272 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:59.057126 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:59.070523 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:59.084957 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:22:59.146293 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 08:22:59.161274 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 08:22:59.167805 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:59.168898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:59.171212 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:22:59.222709 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:22:59.264839 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:22:59.334870 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:22:59.386410 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:59.426522 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(203e7914da3f4494b239c6603513caee): perf score=0.033341
W20240502 08:22:59.445477 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:22:59.584967 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:22:59.606051 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:59.654420 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:59.686254 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:59.740975 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:59.761720 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:22:59.778391 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:22:59.782171 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:22:59.802412 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.376s user 0.096s sys 0.043s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":91487,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":12422,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":164132,"threads_started":2,"wal-append.queue_time_us":59395}
I20240502 08:22:59.803857 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee): 4840 bytes on disk
I20240502 08:22:59.805796 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(203e7914da3f4494b239c6603513caee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 08:22:59.845149 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:22:59.897639 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:22:59.908221 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:22:59.911537 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:22:59.922472 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:22:59.931937 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:22:59.962200 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:00.050660 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:00.057014 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:00.061981 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:23:00.080230 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:23:00.088582 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:00.120891 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:00.144397 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:00.160832 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:00.164444 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.184265 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:00.215545 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:23:00.230914 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:00.255760 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:00.256052 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:00.322126 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:00.325923 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:00.330235 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:00.341212 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:00.348369 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:00.386076 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:00.473188 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:00.482818 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:00.495146 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.507619 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:00.523861 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:00.539423 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.549921 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:00.557077 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.560213 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:00.591713 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:23:00.636178 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:00.649792 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.680517 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:00.706611 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:23:00.741859 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:00.794425 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:00.961966 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:00.975410 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:00.989224 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:01.017727 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:01.027674 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:01.029824 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.062211 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:01.063936 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.064571 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:01.088165 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.095790 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:01.100891 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:01.116226 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:01.181757 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:01.229983 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.247061 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:01.282660 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:01.283046 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 45523us
W20240502 08:23:01.290129 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:01.304975 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:01.329470 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:01.338703 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:01.393754 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:01.495189 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:01.514206 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.615078 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:01.669430 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:23:01.686688 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:01.687296 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:01.697053 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:01.707278 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:01.730659 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:23:01.734123 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:01.752251 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:01.762503 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:01.769663 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:01.771103 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.783650 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:01.805471 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:01.821650 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:01.837677 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:01.849869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:01.860771 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:01.895838 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:02.129235 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:02.145666 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:02.151306 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:02.157266 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:02.188447 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.209769 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:02.211408 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:02.249588 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.255172 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:02.290066 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:02.293056 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:02.300472 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:02.305318 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:02.392400 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:23:02.425590 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:02.449693 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:02.466677 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:02.485766 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:02.534269 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.578140 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:02.598047 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:02.601080 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:02.616394 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.620548 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:02.622352 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.624922 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:02.628320 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:02.645794 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:02.706465 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:02.834189 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:02.955859 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:02.971096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:03.014387 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:03.024175 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:03.043313 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:03.075250 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:03.085862 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:03.093931 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:03.133459 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:03.161073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:03.219722 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:03.313354 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:03.320904 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:03.327129 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:03.338369 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:03.363658 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:03.365391 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:03.370699 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:03.390682 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:03.408092 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:03.408084 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:03.410271 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:03.423030 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:03.473263 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:03.490866 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:03.548096 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:03.600348 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:03.672626 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:03.700306 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:03.711799 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:03.820226 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:03.875070 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:03.909621 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:03.940230 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:03.957158 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:23:03.970026 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:03.990963 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:04.013846 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:04.024703 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:04.070036 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:04.088752 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:23:04.097986 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:04.111152 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:04.123756 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:04.135151 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:04.173161 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 08:23:04.178920 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:23:04.260645 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:04.261445 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:04.284281 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:04.289157 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:04.396497 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:04.397085 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:04.400592 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:04.412811 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:04.438582 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:04.442906 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:04.459744 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:04.478783 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:04.514307 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:04.569626 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:04.647809 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:04.671655 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:04.691211 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 56136.2us
W20240502 08:23:04.694252 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:04.733709 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:04.745114 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:04.746088 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:04.757814 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:04.760645 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:04.812420 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:04.832002 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:04.852595 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:04.871698 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:04.877636 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:04.894073 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling FlushMRSOp(b88c4a1a36424086aad89b7585c17960): perf score=0.033353
W20240502 08:23:04.907835 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:04.948107 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:05.003549 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:05.041800 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:05.047650 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:05.049492 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:05.061772 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:05.061987 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:05.062952 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:05.066516 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:05.078104 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:05.087858 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:05.092756 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:05.179651 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: FlushMRSOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.285s user 0.037s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10423,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":14867,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16055,"threads_started":2,"wal-append.queue_time_us":4604}
I20240502 08:23:05.180974 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960): 4586 bytes on disk
I20240502 08:23:05.199220 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.014s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240502 08:23:05.237385 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:23:05.292411 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:05.341394 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:05.429230 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:05.433068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:05.439232 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:05.455212 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:05.504369 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:05.517477 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:05.584370 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:05.600998 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:05.627534 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:05.627951 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:23:05.703370 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:05.708794 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80): perf score=0.033387
I20240502 08:23:05.746606 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:05.800706 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:23:05.808732 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:05.832242 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:05.837939 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:05.871430 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:05.910374 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:05.947183 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:05.997318 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(852613f14b324b38b5eecc29d4c602cd): perf score=0.033369
I20240502 08:23:06.035727 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.327s user 0.045s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10552,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":3924,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1135,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":39266,"threads_started":2,"wal-append.queue_time_us":19470}
I20240502 08:23:06.037479 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80): 4925 bytes on disk
I20240502 08:23:06.040544 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 08:23:06.097429 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.140231 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:06.148041 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.166465 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:06.188480 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:06.190361 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.232296 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(852613f14b324b38b5eecc29d4c602cd) complete. Timing: real 0.235s user 0.085s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18084,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1032,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":37669,"threads_started":2,"wal-append.queue_time_us":14567}
I20240502 08:23:06.234097 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(852613f14b324b38b5eecc29d4c602cd): 4584 bytes on disk
I20240502 08:23:06.234882 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(852613f14b324b38b5eecc29d4c602cd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4}
I20240502 08:23:06.249293 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:06.255209 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:06.286262 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:06.287226 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.321851 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.343881 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:06.365558 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:06.399533 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:06.407097 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:06.412287 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.427771 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:06.461081 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.466696 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:06.473393 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:06.479701 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:23:06.482167 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:06.520694 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.545859 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf): perf score=0.033355
W20240502 08:23:06.609509 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:06.656689 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.688385 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:06.719944 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.722776 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:06.748178 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:06.770686 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:06.771312 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.788561 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 0.242s user 0.043s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3470,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":7733,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1063,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20882,"threads_started":2,"wal-append.queue_time_us":14171}
I20240502 08:23:06.790169 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf): 4586 bytes on disk
I20240502 08:23:06.796900 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(ebf79c69fd614284ba45fd052cac7ebf) complete. Timing: real 0.006s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":433,"lbm_reads_lt_1ms":4}
W20240502 08:23:06.833241 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:06.843150 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:06.957964 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:06.968504 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:06.989754 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e): perf score=0.033384
W20240502 08:23:07.020759 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:23:07.022258 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:07.025506 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:07.034541 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:07.040022 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:07.057760 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:07.105921 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:07.149341 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:07.206271 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:07.240751 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:07.244647 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:07.258085 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:07.313441 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:07.345875 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:07.347308 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:07.348145 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:07.361361 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:07.398067 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:07.400521 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:07.402293 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: FlushMRSOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.412s user 0.037s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":21527,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":5175,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2625,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":35110,"threads_started":2,"wal-append.queue_time_us":11602}
I20240502 08:23:07.404726 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e): 4840 bytes on disk
I20240502 08:23:07.407392 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 08:23:07.435443 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:07.499850 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:07.531998 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:07.534073 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:07.536303 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:07.586910 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:07.589741 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:07.630474 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:07.658833 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:07.684036 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:07.688262 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:07.719771 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:07.803655 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling FlushMRSOp(d4d933eacd944d84b39f0cb9c45641f8): perf score=0.033387
I20240502 08:23:07.848754 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling FlushMRSOp(182a6f2df91242cfa10f05ef788a19cb): perf score=0.033399
W20240502 08:23:07.892872 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:07.971212 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:08.014253 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:08.042060 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:08.052899 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:08.060603 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:08.104458 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:23:08.105343 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:08.125893 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:08.134626 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:08.142719 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:08.148511 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:08.154999 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:08.196619 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: FlushMRSOp(d4d933eacd944d84b39f0cb9c45641f8) complete. Timing: real 0.392s user 0.114s sys 0.103s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12689,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":16358,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":960,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160336,"threads_started":2,"wal-append.queue_time_us":147616}
I20240502 08:23:08.198128 23517 maintenance_manager.cc:419] P 20b637fbec5444dd91b3af9cdda482f5: Scheduling UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8): 4925 bytes on disk
I20240502 08:23:08.203451 23408 maintenance_manager.cc:643] P 20b637fbec5444dd91b3af9cdda482f5: UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240502 08:23:08.362830 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:08.381906 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: FlushMRSOp(182a6f2df91242cfa10f05ef788a19cb) complete. Timing: real 0.533s user 0.080s sys 0.051s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10606,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1034,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131033,"threads_started":2,"wal-append.queue_time_us":123973}
I20240502 08:23:08.383275 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(182a6f2df91242cfa10f05ef788a19cb): 4925 bytes on disk
I20240502 08:23:08.383882 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(182a6f2df91242cfa10f05ef788a19cb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 08:23:08.522578 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:08.543856 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:08.560946 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:08.594117 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:08.604909 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:08.630548 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:08.647078 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:08.664358 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:08.682816 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:08.694244 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:08.699510 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:08.700400 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:08.709229 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:08.716058 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:08.731199 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:08.756955 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 86612.1us
I20240502 08:23:08.770488 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:08.797243 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:08.894796 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:08.916009 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:08.938222 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:08.953898 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:08.988282 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:08.992846 21616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 90402.3us
W20240502 08:23:08.997323 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:09.009460 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:09.028817 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:09.044378 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:09.048446 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:09.070951 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.070995 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.071113 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:23:09.115176 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.207808 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:23:09.209793 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.246366 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 08:23:09.247522 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.322865 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:09.354588 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:09.363209 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:09.378156 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:09.382454 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:09.386004 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:09.406203 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:09.418653 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:09.424789 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:09.480633 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:09.483377 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:09.485975 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:09.506847 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:09.532768 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:09.586643 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:09.635464 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:09.663004 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:09.684010 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:09.694425 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:09.707976 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:09.735715 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:09.762451 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:09.772614 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:09.776193 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:09.787676 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.817533 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:09.826920 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:09.852124 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:09.869093 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:10.051460 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:10.066802 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:10.069998 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:10.075481 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:10.090744 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:10.117596 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:10.129220 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:10.180074 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:10.199651 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:10.213029 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:10.254596 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:10.352151 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:10.352634 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:23:10.412102 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:10.432600 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:10.442153 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:10.490051 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:10.505429 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:10.527974 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:10.546301 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:10.569869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:10.607021 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:10.642594 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:10.651866 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:10.662871 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:10.703336 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:10.739908 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:10.756085 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:23:10.787667 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:10.808969 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:10.810050 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:10.829815 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:23:10.836462 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:10.856822 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:10.857816 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:10.875036 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:10.933923 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:10.934222 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:10.946848 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:10.990883 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:11.043975 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.133055 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:11.167806 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:11.212441 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:11.212872 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:11.214478 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:11.242871 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:11.276692 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:11.322232 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:11.423115 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:23:11.433964 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.435717 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.511226 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:11.519770 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:11.563189 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:11.566440 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.587893 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:11.595398 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:11.624936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:11.640684 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:11.668068 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:11.670444 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:11.684684 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:11.684851 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:11.726321 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:11.732741 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:11.749529 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:11.770056 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:11.785862 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:11.895023 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.930558 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:11.972456 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:11.992658 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:12.044665 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:12.138254 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.170800 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:23:12.172116 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:12.199347 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.213243 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:12.257375 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:12.257467 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:12.275238 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:12.281711 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:12.312016 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:12.332696 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:12.357357 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:12.391489 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:12.392376 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:12.413239 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:12.446699 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:12.489904 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:12.495293 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:12.538488 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:12.539083 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:12.563068 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:12.612221 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.624441 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:12.647214 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:12.677624 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:12.695506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:12.697716 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:12.711556 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:12.748653 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:12.759956 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:23:12.796058 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:12.851473 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.856971 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.882412 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:12.914713 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:13.005509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.069487 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:13.109081 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:13.122506 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:13.144078 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:13.149696 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:13.197639 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:13.234534 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:13.244650 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.252022 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.306396 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:13.308228 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:13.349531 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:13.351099 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:13.354568 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:13.436949 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:13.456451 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:13.497567 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:13.510679 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:13.604236 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.621152 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:13.623857 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:13.646939 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.652406 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:13.690054 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:13.703729 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:13.780433 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:13.876209 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:13.886380 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:13.890839 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:23:13.893972 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:13.913067 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:13.944803 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:13.949067 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:13.966565 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.021272 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:14.057337 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.126964 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.150940 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.199684 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:14.213110 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.257083 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:23:14.260903 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:23:14.309585 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:14.362339 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:14.420421 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:14.421406 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:14.449805 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:14.454412 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:14.458688 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:14.459945 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:14.472440 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:14.501869 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:14.502514 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:14.514283 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:14.531939 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:14.538501 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:14.584825 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:14.592532 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:14.603196 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:14.607635 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:14.610411 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:14.626117 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:14.644122 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:14.671561 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:14.687183 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:14.689713 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:14.719997 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:14.737746 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:14.786742 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:14.849000 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:14.857617 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:14.876646 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:14.887128 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:14.898377 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:14.908819 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:14.911043 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:14.926752 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:14.959041 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:14.959290 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:14.970454 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:15.109905 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:15.208202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:15.228513 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:15.244791 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:15.252715 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:15.272104 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:15.284368 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:15.313617 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 81670.2us
W20240502 08:23:15.316079 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:15.372570 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:15.408061 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:15.410825 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:15.493510 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:23:15.533530 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:15.542613 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:23:15.554935 23405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 144106us
W20240502 08:23:15.588219 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:15.611675 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:15.669694 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:15.698761 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:15.741799 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:15.743172 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:15.836118 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:15.864259 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:15.868463 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:15.870450 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:15.890548 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:15.934268 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:15.998153 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:23:16.029472 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:16.124949 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:16.178009 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:16.224952 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:16.268959 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:16.278497 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:16.282588 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:16.288215 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 54313us
W20240502 08:23:16.295405 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:23:16.298172 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:16.301689 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:16.319597 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:16.323558 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:16.347021 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:16.370484 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:16.376236 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:16.398367 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:16.402892 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:16.404207 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:16.413578 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:16.414996 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:16.418627 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:16.525583 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:16.537458 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:16.545845 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:16.550858 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:16.561158 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:16.564291 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:16.582926 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 69479.9us
I20240502 08:23:16.584215 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:16.588518 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:16.590703 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:16.653002 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:16.655371 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:16.667455 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:16.740532 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:16.769869 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:16.772047 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:16.783951 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:16.864645 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:16.951754 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:17.075546 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:17.104918 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:17.108448 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:17.114085 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:17.135082 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:17.142930 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:23:17.191272 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:17.200551 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:17.205543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:17.223733 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:17.240926 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:17.283385 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:17.285343 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:17.300113 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:17.308136 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:17.313491 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:23:17.314597 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.364877 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:17.373435 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.383072 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.422683 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:17.509542 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:17.519480 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:17.538275 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:17.557937 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:17.593829 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:17.601471 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:17.623440 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:17.651103 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:17.721817 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.728631 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 64229us
W20240502 08:23:17.782596 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:17.897845 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:17.905580 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:17.905855 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.940914 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:17.951829 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:17.979930 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:17.988176 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:18.030519 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:18.067673 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:18.089532 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:18.095062 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:18.122555 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:18.123834 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:18.187369 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:18.203387 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:18.206869 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:18.261772 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:18.267321 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:18.276722 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:18.298919 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:18.373920 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:18.374102 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:18.474710 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:18.616732 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:18.658820 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:18.678728 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:18.703065 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:18.708580 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:18.712918 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:18.722885 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:18.741580 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:18.770433 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:18.930974 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:18.953425 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:23:18.957465 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:18.978899 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:19.015834 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:19.043849 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:19.046902 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:19.051084 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:19.051648 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:19.070931 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:19.083927 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:19.095641 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:19.122576 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:19.127985 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:19.139631 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:19.163864 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:19.180636 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:19.183485 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:19.214886 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:19.255204 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:19.386361 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 08:23:19.393850 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:19.421211 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 08:23:19.427532 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:19.427793 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:19.492691 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:19.517354 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:19.526337 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:19.533577 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:19.545507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:19.564848 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:19.568387 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:19.616883 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:19.640444 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:19.651043 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:19.673411 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:23:19.721573 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 48181.6us
W20240502 08:23:19.733173 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:19.888702 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:19.911814 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:19.949075 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:19.977378 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:19.979511 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:19.989697 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:20.006911 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:20.017894 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:20.018785 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:20.043460 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:20.072461 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:20.091145 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:20.112619 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:20.141616 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.144747 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.170832 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:20.177763 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:20.184259 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:20.185326 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:20.187584 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:20.195753 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:20.209553 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:20.212201 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:20.251017 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:20.290184 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:20.301676 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:20.324383 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:20.463734 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:20.464026 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.478946 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:20.486400 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:20.504334 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:20.529228 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:20.530427 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:20.535694 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:20.548213 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:23:20.578486 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:23:20.636062 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:20.637777 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:20.705612 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.728521 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.846148 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:20.881853 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:20.982041 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:21.002760 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:21.041093 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:21.094362 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:21.098704 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:21.156662 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:21.193737 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:21.214201 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:21.322794 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:21.365005 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:23:21.383456 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.398906 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.432610 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:21.433431 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:21.436570 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:21.438745 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:23:21.446581 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:21.456686 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:21.496929 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.506034 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.508352 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:21.511147 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:21.526124 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:21.542846 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:21.550381 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:21.554234 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:21.557421 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:21.564914 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:21.587713 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.592276 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:21.669643 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:21.690310 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:21.719534 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:21.720034 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:21.800302 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:21.828280 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:21.850481 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:21.856750 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:21.939399 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:22.010928 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:22.057574 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:22.092052 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:22.098902 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:22.141882 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:22.145651 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:22.171870 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:22.187279 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:22.193279 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:22.205713 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:22.333403 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:22.417250 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:23:22.420387 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:22.475553 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:22.505812 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:22.509701 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:22.522439 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:22.534677 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:22.543485 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:22.545239 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:22.585150 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:22.604210 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:22.618292 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:22.619212 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:22.632129 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:22.658675 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:22.689184 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:22.725956 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:22.756098 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:22.759339 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:22.806182 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:22.878535 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:22.903457 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:22.943368 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.013530 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:23.080673 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:23.081923 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:23.086344 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:23.113832 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:23.126562 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:23.161402 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:23.171866 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:23.200309 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:23.202441 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:23.219156 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.290614 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.373000 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:23.381033 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:23.391692 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:23.413257 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:23.416260 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:23.441219 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:23.479420 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:23.508760 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.563158 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:23:23.606732 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:23.746620 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:23.797262 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:23.807678 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:23.844123 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:23.853276 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:23.889089 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:23.910691 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.926524 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:23.941422 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:23.958086 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:23.985713 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:23:24.060777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:24.072531 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:24.076630 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:24.110713 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:24.120136 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:24.156769 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:24.171066 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:24.194600 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:24.210703 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:24.221262 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:24.325264 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:24.338784 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:24.397886 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:23:24.399350 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:24.469259 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 08:23:24.516808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:24.517379 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:24.537243 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:24.568368 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:24.636651 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:24.652446 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:24.653663 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:24.664700 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:24.690994 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:24.691534 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:24.695140 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:24.695981 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:24.709901 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:24.733621 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:24.735133 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:24.752821 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:24.760720 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:24.776600 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:24.787645 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:24.806490 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:24.861114 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:24.876950 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:24.898139 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:24.903255 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:24.907687 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:24.925041 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:23:24.925186 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:24.943980 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:24.986984 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 61951.9us
W20240502 08:23:25.045768 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.048077 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.148295 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:25.189520 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.218495 21485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 59280.8us
W20240502 08:23:25.243937 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:25.263931 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:25.373929 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:25.376570 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:25.402446 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:25.402865 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:25.426576 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:25.438593 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:25.447878 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:25.492355 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:25.507174 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:25.562378 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:23:25.566071 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:25.582253 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:25.602985 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:25.608803 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:25.612850 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:25.639120 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:25.644940 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:25.683452 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.766616 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:25.824229 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.834867 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:25.953466 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:23:26.015625 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:26.086441 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:26.106876 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:26.117070 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:26.129928 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 57399.2us
I20240502 08:23:26.176893 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:26.212420 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:26.225379 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:26.225732 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:26.357421 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:26.401901 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:26.409226 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:26.427832 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:26.438198 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:26.444247 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:26.454536 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:26.458016 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:26.477087 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:26.494149 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:26.495157 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:23:26.505448 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:26.508497 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:26.515291 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:26.624936 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:23:26.655552 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:26.659862 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:26.681625 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:26.688362 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:26.717443 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:26.779577 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:26.815071 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:26.823608 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:26.922096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:26.952795 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:26.970196 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:26.973062 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:27.006527 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:27.106794 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:27.109198 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:27.128176 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:27.132627 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:27.158061 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:27.171380 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.192636 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:27.233737 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.322566 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:27.410153 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:27.422629 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:27.439800 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.462023 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:27.464485 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:27.464622 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:27.465206 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:27.483526 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:27.487576 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:27.522233 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:23:27.523186 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:27.535207 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:27.541911 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:27.546609 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:27.561232 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:27.599020 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:27.622066 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:27.673115 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.683450 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.698208 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:27.719434 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:27.791539 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:27.921857 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:27.924392 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:27.956516 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:27.987862 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:27.997467 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:27.998306 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:28.062759 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:28.081043 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:28.116374 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:28.218413 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:28.232430 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:28.232753 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:28.249580 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:28.260489 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:28.293481 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:28.294282 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:28.338064 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:28.376808 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:28.395596 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:28.541436 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:28.548418 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:28.550940 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:28.553580 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:28.561731 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:28.565462 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:23:28.587082 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:28.618829 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:28.804090 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:28.858278 23407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 54009.1us
I20240502 08:23:28.886507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:28.891821 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:28.924500 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:28.936502 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:28.969478 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:28.986335 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:28.988723 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:29.010139 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:29.013378 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:29.030704 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.041579 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.057679 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:29.076068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:29.079555 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:29.094703 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:29.111685 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:29.139931 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:23:29.143224 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:29.225152 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:29.324285 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:29.347959 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:29.411476 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:29.476168 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240502 08:23:29.506847 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.541419 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:29.552335 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:29.579039 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 08:23:29.620507 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:29.622489 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:29.642324 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:29.657591 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:29.658780 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:29.665977 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.676327 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.705915 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:29.705957 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:29.742725 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:29.762583 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:29.792567 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:29.836567 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:29.866833 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:29.895406 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:29.911489 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:29.939766 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:29.941149 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:29.943071 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:29.975210 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:29.977555 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.006981 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.021569 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:30.038441 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:30.068291 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:30.104079 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:30.118813 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:30.167750 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.217661 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:30.237036 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:30.283349 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:30.304359 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:30.324671 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.342806 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:23:30.363003 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:30.363443 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:30.367835 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:30.373612 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:30.396762 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:30.461670 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:30.503535 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:30.503629 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:30.510985 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:30.533540 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:30.537122 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:30.560462 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.570787 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:23:30.601220 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.614790 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.702242 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.742389 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:30.770583 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:30.886658 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:23:30.911670 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:30.982820 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.057420 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:31.063714 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:31.068698 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:31.117439 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:31.120023 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:31.218969 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:31.222638 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:31.240056 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:31.290644 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.297678 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.302146 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:31.324564 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:31.337404 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:31.343818 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:31.346709 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:31.373371 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:31.379099 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:31.383775 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:31.384850 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:31.401047 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:31.436203 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:31.451838 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.600441 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.603549 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:31.615571 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:31.626855 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:31.638819 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:31.657270 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:31.659749 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:31.665026 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:31.678725 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:31.697145 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:23:31.757877 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:31.796617 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:31.802807 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:23:31.943575 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.969460 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:31.979526 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:31.981165 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:31.988940 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:31.989356 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:32.032739 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:32.062630 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:32.134621 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:32.146900 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:32.232852 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:32.253952 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:32.254146 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:32.320892 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:32.352196 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:32.393436 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:32.467108 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:32.477977 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:32.498029 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:32.507431 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:32.518064 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:32.564016 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:32.564580 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:32.592756 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:32.617297 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:23:32.632406 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:32.729352 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:32.782238 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:32.832350 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:32.839553 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:32.840682 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:32.842092 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:32.875341 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:32.898164 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:32.914980 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:32.989823 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:32.998814 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:33.013513 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:33.111627 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:33.166050 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:33.172015 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:33.172096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:33.197309 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:33.205070 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:33.212011 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:33.215152 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:33.223861 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:33.224143 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:33.261718 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:33.275336 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:33.294879 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:33.295948 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:33.312631 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:33.319432 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:33.348242 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:33.456192 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:23:33.469774 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:33.470750 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:33.493017 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:33.497995 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:33.508493 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:33.551345 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:33.558569 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:33.694504 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:33.720064 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:33.720979 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:33.731645 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:33.754222 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:33.802887 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:33.810807 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:33.825903 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:33.829044 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:33.856860 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:33.943751 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:34.034117 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:34.039765 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:34.052110 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:34.071513 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:34.100682 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:34.104000 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:34.105310 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 61924.7us
I20240502 08:23:34.134606 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:34.143024 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:34.180085 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.359742 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:23:34.388154 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.488389 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.507360 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:23:34.512686 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:34.527585 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.554301 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:34.574965 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.632851 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:34.693666 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 08:23:34.791081 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:34.794687 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:34.815536 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:34.831530 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:34.852708 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:34.862905 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:34.875757 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:23:34.892295 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:34.893535 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:34.895134 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:34.902740 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:34.916990 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:34.949700 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:34.972674 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:34.985749 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:35.005031 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:35.051553 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:35.051674 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:35.084246 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:35.121881 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:35.123781 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:35.137826 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:35.143193 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:35.160337 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:35.163225 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:35.165769 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:35.182750 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:35.193639 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:35.206283 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:35.209344 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:35.218480 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:35.241267 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:35.261667 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:35.270987 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:35.295076 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:35.322038 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:35.335860 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:35.337718 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:35.370023 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:35.379858 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:35.383608 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.432853 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.493567 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.533238 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.595630 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:35.597810 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.650833 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:35.663197 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:35.779599 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.877802 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:35.893213 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:23:36.009572 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:36.016646 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:36.041643 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:36.061944 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:36.063778 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:36.108872 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:36.120313 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:36.140795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:36.147678 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:36.160382 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:36.171489 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:36.178735 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:36.205255 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:36.221441 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:36.221554 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:36.270800 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:36.278430 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:36.345161 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:36.384492 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:36.401234 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:36.474313 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:36.529155 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:36.537240 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:36.552603 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:36.553856 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:36.582616 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:36.612514 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:36.727169 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:36.739499 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:36.741948 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:36.761174 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:36.767263 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 43840.1us
I20240502 08:23:36.792578 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:36.816907 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:36.839876 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:23:36.870085 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:36.873466 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:36.873991 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:36.931391 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:36.946275 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:23:36.960158 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:36.967556 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:36.997149 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:37.017652 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:37.028302 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:37.031061 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:37.096848 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:37.190568 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:37.221252 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:37.283035 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:37.292373 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:37.296290 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:37.302755 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:37.335824 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:37.349511 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:37.368145 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:37.388263 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:37.503314 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:37.522120 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:37.548213 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:37.554433 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:37.556470 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:37.571591 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:37.580260 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:37.637013 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:37.641187 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:37.641907 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:37.642706 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:37.669184 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:37.797601 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:37.820466 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:37.824963 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:23:37.826053 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:37.842320 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:37.863423 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:37.912092 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:38.118364 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:38.146929 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:38.185420 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.185787 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:38.221448 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:38.225728 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:38.249603 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:38.319577 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:38.331955 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:38.354915 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.367089 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:38.382194 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:38.407665 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:38.409358 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:38.425774 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:38.434373 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:23:38.443074 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.449270 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:38.454494 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:38.465310 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:23:38.495946 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:38.531919 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:38.535045 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:38.574043 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.576112 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.577931 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.578656 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:38.601002 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:38.602766 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:38.603495 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:38.643462 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:38.678822 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:38.678970 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:38.796603 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:38.805212 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:38.878463 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:38.912773 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:39.020812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:39.022799 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:39.031682 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:39.033401 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:39.055723 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:39.056563 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:39.103899 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.196380 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:39.224781 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:39.229831 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:23:39.245860 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:39.282598 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.407234 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.458053 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.487380 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.542738 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 08:23:39.595485 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:39.600399 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:39.626176 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:39.654043 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:39.672482 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:39.711812 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:39.723771 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:39.735435 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 08:23:39.740805 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:39.788741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:39.794366 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:39.796404 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:39.829223 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:39.836982 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:39.867884 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:39.882337 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:39.882870 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:39.910799 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:39.914923 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:39.917681 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:39.959261 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:39.965389 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:40.027068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:40.028101 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:40.045118 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:40.062167 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:40.092509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:40.134364 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:40.147096 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:40.161584 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:40.174686 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:40.209550 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:40.245539 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:40.248644 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:40.258126 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:40.259085 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:40.287801 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:40.316747 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:40.374593 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:40.448216 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:40.502323 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:40.508600 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:40.520174 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:40.529295 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:40.537725 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:40.539320 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:40.561594 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:40.582582 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:40.759832 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:40.764664 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:40.768013 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:40.883469 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:40.901013 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:41.032207 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:41.078573 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:23:41.103132 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:41.117753 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:41.148563 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:41.162567 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:41.193239 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:41.201623 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:41.203565 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:41.207293 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:41.253031 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:41.259653 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.264729 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:41.289600 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 66467.7us
W20240502 08:23:41.295711 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:41.306218 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.318727 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:41.321774 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:41.361564 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:41.379832 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:41.381510 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.539417 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:41.558030 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:41.558861 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.600503 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:41.620782 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:23:41.640026 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:41.641832 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.655069 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:41.678062 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:41.706270 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:41.712213 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:41.729645 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:41.757680 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:41.780155 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:41.796890 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:41.807160 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 72034.4us
I20240502 08:23:41.830572 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:41.833947 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:41.838632 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:41.859514 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:41.883880 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:41.915772 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:41.969766 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:42.011659 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:42.033146 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:42.048101 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:42.052659 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:42.109385 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:23:42.119001 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:42.120071 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:42.339661 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:42.376643 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:42.380276 21485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 46061.6us
W20240502 08:23:42.431972 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:42.436782 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:42.450656 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:42.450685 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:42.491768 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:42.507704 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:42.526518 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:42.544260 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:42.560148 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:42.584971 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:42.596765 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:42.624848 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:42.645125 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:42.663368 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:42.665632 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:42.688055 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:42.689783 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:42.723383 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:42.748683 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:42.899883 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:42.908532 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:42.923343 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:23:42.924535 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:42.974858 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:43.005952 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:43.049490 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:43.091127 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:43.120137 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:43.140319 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:43.174324 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:43.204367 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:43.219182 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:43.235595 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:43.249629 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:43.272209 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:43.275427 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 47836.6us
W20240502 08:23:43.278455 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:43.290949 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:43.307914 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:43.312292 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:43.396456 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:43.402961 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:43.410898 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:23:43.443171 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:43.469121 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:43.469574 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:43.490851 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:43.505885 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:43.525777 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:43.527984 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:43.564282 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:43.695935 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:43.757216 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:43.765347 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:43.803682 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:43.828492 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:43.831077 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:43.875195 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:43.925303 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:43.971657 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:43.981330 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:43.995951 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:44.057425 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:44.074005 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:44.078787 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:44.088637 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:44.125993 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:44.136674 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:44.153044 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:44.215700 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:23:44.324056 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:44.382769 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:44.409111 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:44.424795 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:44.451042 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:44.451473 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:44.486106 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:44.550640 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 08:23:44.569137 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:44.572628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:44.603744 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:44.646123 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:44.671041 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:44.697644 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:44.705261 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:44.711282 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:44.728567 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:44.748252 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:44.785178 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:44.791556 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:44.806771 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:44.820977 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:44.847868 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:44.852401 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 08:23:44.933923 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:44.950381 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:44.993283 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:45.017733 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:45.021979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:45.026799 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:45.073346 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:45.073765 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:45.092564 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:45.096169 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:45.124768 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:45.154249 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:45.161813 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:45.181454 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:45.207640 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:23:45.223498 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:45.237633 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:45.255686 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:45.290063 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:45.304853 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:45.334735 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:45.423081 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:45.424913 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:45.447363 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:45.514506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:45.574153 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:45.604910 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 58221.2us
W20240502 08:23:45.625198 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:45.654912 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:45.655741 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:45.660888 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:45.677060 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:45.761051 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:45.848403 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:45.863674 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:45.872822 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:45.888435 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:45.902920 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:46.011699 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:46.070808 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:46.095942 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:46.128154 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:23:46.189561 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:46.218943 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:46.229818 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:46.261778 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:46.264873 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:46.271812 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:46.290875 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:46.316526 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:46.324273 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.329449 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:46.375841 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:46.384603 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:46.402603 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:46.473918 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:46.487979 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.490936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.492036 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:46.498975 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:46.509860 21531 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.510313 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.517624 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:46.535290 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:46.606326 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:46.669013 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:23:46.783624 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:46.808266 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:46.886837 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:46.900524 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:46.933581 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:46.960582 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:47.012935 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:47.024394 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:47.040333 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:47.070056 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:47.133889 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:23:47.134806 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:47.198004 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:47.199828 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:47.200896 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:47.251951 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:47.253659 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:47.281167 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:47.407001 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:47.449187 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:47.449829 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:47.465286 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:47.493413 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:47.507699 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:47.521760 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:47.525352 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:47.539705 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:47.539803 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:47.542605 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:47.547582 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:47.550104 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:47.556545 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:47.590183 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:47.596150 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:47.630702 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:47.636852 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:47.643704 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:47.688526 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:47.691226 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:47.845069 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:47.858444 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:47.872920 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:47.876242 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:47.889114 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:47.986796 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:47.989835 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:23:48.001271 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.008890 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:48.015668 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 69511.3us
I20240502 08:23:48.034793 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:48.037977 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:48.040859 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:48.051836 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.063205 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:48.085531 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.142652 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.187162 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.241990 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:48.333735 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.381443 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:48.440333 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.465072 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:48.469106 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:48.472957 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:48.495401 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.505427 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:48.508813 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:48.599846 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:23:48.635321 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:48.698854 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.722147 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:48.753006 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:48.785782 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.804484 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:48.821095 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:48.830916 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:48.844676 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:48.865304 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 174549us
I20240502 08:23:48.893270 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:48.919293 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:49.007037 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:49.027916 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.033937 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:49.036863 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:49.063694 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:49.075536 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:49.110016 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:49.115294 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:49.155557 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:49.209805 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:49.270152 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.304085 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:49.342850 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:49.364614 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:49.400179 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:49.403306 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:49.408685 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:49.429200 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:49.455543 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:49.476163 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:49.478894 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:49.507040 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:49.526496 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:49.545940 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:23:49.559571 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:49.571069 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:49.573627 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:49.600868 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:23:49.653618 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.679482 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.725737 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:49.730669 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:49.777894 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.818270 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:49.921334 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:49.927274 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:23:49.928877 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:49.972446 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:50.023097 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:50.032828 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:50.055488 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:50.060009 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:50.065740 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:50.078915 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:50.107652 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:50.119271 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:50.126935 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:50.186758 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:50.289988 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:50.325037 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:50.332985 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:50.386860 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:50.430370 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:50.459283 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:50.464061 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:50.499289 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:50.511227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:50.606423 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:50.646059 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:50.680737 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:50.821327 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:50.830695 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:50.849821 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:50.879174 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:50.903419 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:50.913492 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:50.919108 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:50.950805 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:50.966171 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:50.969154 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:50.976233 21663 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:50.976461 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:51.017191 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:51.029506 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:51.030884 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:51.064957 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:51.068315 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:51.145862 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:51.160749 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:51.169703 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:51.172835 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:23:51.208879 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:51.209483 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:51.228559 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:51.241094 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:51.282685 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:23:51.305639 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:51.326543 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:51.357249 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:51.454082 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:51.465698 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:51.525439 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:51.611045 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:51.627343 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:51.631882 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:51.652956 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:51.692152 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:51.748399 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:51.773852 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:51.797024 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:51.797389 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:51.857553 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:51.887861 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:51.898763 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:51.949855 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:51.977838 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:52.012353 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:52.014159 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:52.019382 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:52.036407 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:52.070871 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:52.097597 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:23:52.114779 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:52.133098 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:52.144507 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:52.153832 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:52.154449 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:52.178437 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:52.188367 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:52.189440 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:52.203612 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:23:52.280032 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:52.288677 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:52.301244 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:52.340370 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:52.535985 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:52.573941 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:52.578316 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:52.591305 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:52.619511 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:52.652348 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:52.659260 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:52.675962 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 58910us
I20240502 08:23:52.677734 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:52.724948 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:52.738739 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:52.788532 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:52.839957 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:52.841141 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:52.859560 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:52.883618 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:52.904348 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:52.909754 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:52.926364 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:52.926781 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:52.995566 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:53.022711 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:53.026464 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:53.030457 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:53.063985 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:53.105283 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:53.109545 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:53.113816 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:53.136803 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:23:53.192492 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.192625 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.200938 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:53.357097 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:53.393805 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:53.412845 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:53.418687 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:53.427561 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:53.433604 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:53.453562 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:53.468485 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:53.473659 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:53.525079 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:23:53.544273 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:53.550153 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:53.550576 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:53.588253 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:53.595170 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:53.601850 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 58115.7us
W20240502 08:23:53.610777 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.618196 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.630604 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:53.638347 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:53.714485 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.800994 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:53.878291 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:53.987092 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:53.992570 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:54.031277 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:54.057291 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:54.072214 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:54.095130 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:54.114615 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:54.162360 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:54.198056 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:54.240478 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:54.261142 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:54.318094 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:54.337090 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:54.339433 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:54.405540 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:54.405997 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:54.434634 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:54.442399 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:54.446485 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:54.470201 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:54.498740 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:54.505492 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:54.508749 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:54.524140 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:23:54.525291 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:54.564884 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:54.619657 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 08:23:54.680122 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:54.694097 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:54.738121 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:54.807963 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:54.854110 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:54.878983 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:54.889802 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:54.909371 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:54.910315 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:54.912000 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:54.930358 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:54.957252 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:55.006884 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:55.025306 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 08:23:55.026433 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:55.081929 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:55.135110 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:55.153898 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:23:55.206786 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:55.211796 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:55.213192 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:55.217262 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:55.266822 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:55.269109 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:55.281497 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:55.309194 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:55.375921 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:55.459228 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:55.553771 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:55.562100 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:55.567320 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:55.620900 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:55.646867 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:55.654768 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:23:55.679119 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:55.679759 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:55.696911 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:55.784543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:55.790437 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:55.825947 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:55.859210 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:55.864926 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:55.877573 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:55.884029 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:55.899070 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:55.934499 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:55.948155 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:55.964571 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:55.986791 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:55.995796 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:56.026441 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:56.033509 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:56.055610 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:56.070645 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:23:56.071750 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:56.089593 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:56.129979 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 50062.3us
W20240502 08:23:56.131839 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:56.146855 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:56.157948 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:56.189262 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:56.189663 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:23:56.348794 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:56.355854 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:56.363338 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:56.368515 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:56.387470 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:56.405963 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:56.447530 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:56.453501 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:56.671296 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:56.692516 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:56.695338 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:23:56.756402 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:56.796047 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:56.805104 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:56.808138 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:56.834604 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:56.847051 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:56.870527 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:56.943730 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:57.005375 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:57.016577 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:57.126850 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:23:57.153062 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:57.207842 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:57.268601 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:57.280936 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:57.287698 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:57.291507 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:23:57.310372 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:57.333848 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:57.345103 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:57.360890 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:57.375265 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:57.378240 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:57.403474 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:57.447080 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:57.477327 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:57.498410 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:57.527383 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:57.535459 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:57.536464 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:57.539072 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:57.544880 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:57.583671 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:57.586707 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:57.605178 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:57.625304 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:57.632259 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:57.673427 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:57.674324 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:23:57.682018 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:57.783388 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:57.874342 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:57.897979 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:57.924052 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:57.949546 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:57.949967 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:57.986322 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:58.010322 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:58.031345 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:58.033677 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:58.062606 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:58.132942 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:58.159564 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:58.188264 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:58.192862 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:58.272505 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:23:58.273475 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:58.278134 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:58.305999 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:58.321007 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:58.328832 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:58.387508 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:58.465883 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:58.485188 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:58.583993 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:58.607913 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:58.619235 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:58.639369 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:58.674561 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:58.683068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:58.685299 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:58.701184 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:58.742218 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:58.770923 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:58.828994 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:23:58.875685 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:23:58.897181 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:58.944339 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:58.994050 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:59.005986 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:23:59.006764 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:59.013389 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:59.040524 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:59.063560 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:59.081127 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:59.086954 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:23:59.107976 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:59.147341 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:59.221912 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:59.233178 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:59.258116 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:59.334556 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:59.344092 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:23:59.360836 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:59.361258 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:59.376549 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:59.405843 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:23:59.420154 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:23:59.473727 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:59.481038 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:23:59.488503 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:59.515763 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:23:59.547830 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:59.639973 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 08:23:59.652294 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:23:59.683390 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:23:59.702864 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:23:59.715164 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:23:59.718596 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:23:59.770813 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:23:59.804059 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:23:59.827610 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:59.828603 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:59.834883 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:23:59.862141 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:23:59.913093 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.053068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:00.056495 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:24:00.066848 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:24:00.071403 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.081444 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:00.087054 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.097406 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:00.120086 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:00.153741 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:24:00.207661 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.225245 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:00.263383 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.283973 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:00.294582 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:00.297566 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:00.338909 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.347620 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:00.352900 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:00.403743 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:24:00.433205 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:00.449330 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:00.461977 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:00.475855 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.562458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.587234 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:00.592078 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:00.597368 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:00.644115 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:00.661692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.695552 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:00.727823 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.870729 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:00.874265 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:00.885033 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:00.887127 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:00.899900 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:00.901749 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:00.916275 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:00.969825 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:00.983614 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:00.998982 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:01.006824 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:24:01.080590 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:24:01.082158 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:01.106009 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:01.209152 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:01.272279 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:01.295559 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:01.313297 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:01.366091 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:01.374809 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:24:01.393903 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:01.410420 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:01.412326 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:01.420778 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:01.423516 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:01.427033 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:01.452977 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:01.470521 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:01.481122 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:01.483105 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:01.533936 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:01.535212 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:01.645126 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:01.725894 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:01.786813 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:01.810537 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:01.852239 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:01.859822 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:01.877460 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:01.903566 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:01.913625 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:01.919613 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:01.951458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:01.987684 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:24:01.990088 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:01.994342 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:02.022581 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:02.041765 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:02.113045 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:02.168432 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:02.179946 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:02.186129 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:02.212666 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 53624.2us
W20240502 08:24:02.255527 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:02.303002 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:02.308779 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:02.331681 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:02.332445 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:02.353693 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:02.399657 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:02.423894 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:02.472043 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:02.491844 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:02.532505 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:02.577240 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:02.603421 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:24:02.620708 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:02.621820 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:02.646997 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:02.652391 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:02.693696 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:02.700647 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:02.706497 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:02.748541 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:02.786545 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:02.796923 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:02.818686 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:02.871836 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:02.880707 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:02.886111 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:02.903277 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:02.904778 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:02.925213 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:02.954813 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:02.986080 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:03.040077 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:03.040238 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:03.183537 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:03.219310 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:03.249711 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:03.252825 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:03.272572 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:03.293659 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:03.298929 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:03.317721 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:03.336864 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:03.353870 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:03.387929 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:24:03.411487 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:03.445384 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:03.448547 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:03.477254 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:03.478267 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:03.505533 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:03.506374 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:03.522835 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:03.538380 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:03.544495 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:03.633937 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:24:03.644420 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:03.653666 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:03.781141 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:03.781284 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:03.803033 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:03.836741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:03.852820 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:03.891258 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:03.917979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:03.929215 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:03.967528 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:03.982506 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:04.016419 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:04.026583 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:04.047799 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:04.085531 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:24:04.106688 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:04.194504 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:04.212919 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:04.223412 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:04.228781 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:04.236231 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:04.247130 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:04.314682 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:04.322875 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:04.335423 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:04.353543 23405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 137788us
I20240502 08:24:04.374092 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:04.404372 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:04.427165 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:04.608085 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:04.611501 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:04.635038 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:04.661221 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:24:04.694768 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:04.710352 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:04.712093 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:04.736037 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:04.758970 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:04.774621 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:04.779176 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:04.789876 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:04.978514 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.062886 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:24:05.087204 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:05.097478 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:05.114738 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:05.140587 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:05.143895 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:05.162070 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:05.163623 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:05.171099 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.224299 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:05.224429 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:05.238207 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.269012 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.290881 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:05.297582 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 50943.2us
W20240502 08:24:05.314642 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:05.361193 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:05.402428 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:05.511445 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:05.517257 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:05.575278 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:05.576392 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:05.579659 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:05.587175 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:05.611531 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:05.633023 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:05.633994 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:05.662163 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:05.667326 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:05.680240 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:05.720906 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:05.726810 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:05.773166 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:05.773620 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:05.786414 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 79622.2us
I20240502 08:24:05.831310 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:05.838791 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:05.854756 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:05.861583 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:05.862706 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:05.864987 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.872642 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:24:05.872998 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:05.895367 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:05.899333 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:05.968554 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:06.041004 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:06.090416 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:06.242656 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:06.299592 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:06.301282 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:06.311173 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:06.321410 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:06.321851 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:06.324575 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:06.331197 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:06.476200 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:06.503348 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:24:06.508734 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:06.712240 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:24:06.750448 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:06.768003 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:06.795154 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:24:06.832319 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:06.846668 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:06.850934 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:06.885530 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:06.909763 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:06.918282 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:06.951637 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:06.998265 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.062955 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:07.078431 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:07.087530 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:07.107568 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:07.142072 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:07.184041 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:07.226042 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:07.231523 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:07.236300 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:07.242141 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:07.325349 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 135967us
I20240502 08:24:07.336695 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:07.338230 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:07.347047 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:07.356053 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:07.368871 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:07.376891 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:07.384446 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:07.510807 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:07.527590 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:07.533068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:07.557230 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.558606 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:07.573407 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:07.579773 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:07.587556 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.599828 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.606787 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:24:07.615912 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:07.642292 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:07.653347 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:07.673575 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:24:07.679450 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.717002 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:07.779321 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:07.853111 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:07.907178 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:07.946017 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:07.946259 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:07.994956 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:07.996520 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:08.043365 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:08.065740 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:08.066777 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:08.076663 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 48160.3us
I20240502 08:24:08.084582 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:08.096210 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:08.201035 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:08.345697 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:08.362726 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.399027 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.400207 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.421743 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:08.423584 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.460872 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:08.493474 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:08.504734 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:08.547473 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.599571 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:08.618235 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.638139 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.646485 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:08.656535 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.666749 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:08.681568 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:08.682641 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:24:08.712860 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:08.719427 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:08.732062 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.756556 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:08.758690 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:08.841173 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:08.841436 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:08.860072 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:08.862020 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.870599 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:08.875684 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:08.905174 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:08.918136 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:08.919004 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:08.941026 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:08.961191 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:08.972468 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:08.996740 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:09.063519 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:09.222538 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:09.224123 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:24:09.244552 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:09.296645 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:09.323555 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:09.323926 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:09.338856 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:09.395855 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:09.420516 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:09.524539 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:09.634936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:09.646648 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:09.698523 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:09.701786 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:09.754021 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:09.755084 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:09.780571 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:09.791484 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 08:24:09.823890 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:09.867913 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:09.877576 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:09.965929 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:09.979074 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:10.085347 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:10.103516 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:10.120329 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:10.153074 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 08:24:10.154476 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:10.163839 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:10.168804 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:10.199690 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:10.200227 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:10.230895 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:10.249819 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:10.252751 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:10.260943 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:10.260988 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:10.323514 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:10.374537 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:10.393302 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:10.420912 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:24:10.476676 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:10.496731 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:10.505769 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:10.506208 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:10.509913 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:10.513132 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:10.549419 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 66760.4us
I20240502 08:24:10.554001 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:10.569947 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:10.571486 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:10.612191 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:10.663156 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:10.680507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:10.681262 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:10.706192 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:10.715404 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:10.750576 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:10.792034 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:10.796072 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:10.845347 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:10.899636 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:10.909181 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:10.915169 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:10.919993 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:10.935671 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:10.956952 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.040613 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.081069 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.112636 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.182708 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.198143 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:11.212635 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:11.317572 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 52045.7us
I20240502 08:24:11.380975 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:11.381793 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:11.387303 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:11.408507 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:11.429118 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:11.432188 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:11.432644 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:11.491793 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:11.559471 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:11.594758 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:11.630375 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:11.650722 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:11.672106 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:11.684820 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:11.728557 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:11.743517 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:11.872376 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:24:11.905448 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:24:11.933457 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:24:12.024528 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:12.033632 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:12.058202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:12.085978 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:12.133889 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:12.162068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:12.162772 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:12.196806 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:12.245615 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:12.252048 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:12.254890 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:12.260324 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:12.261683 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:12.262878 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:12.290964 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:12.292513 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:12.327713 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:12.410146 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:12.421823 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:12.433694 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:12.447762 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:12.462287 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:12.486001 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:12.489185 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:12.526883 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:12.644109 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:12.661186 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:12.672598 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:12.677515 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:12.728534 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:24:12.730001 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:12.763064 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:12.821357 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:12.842924 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:12.846757 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:12.874485 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:12.902287 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:12.912619 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:12.954005 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:12.992101 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:13.004554 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:13.011752 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:13.021330 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:13.060566 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:13.072470 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:13.088068 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:13.092130 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:13.106159 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:13.128042 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:13.132032 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:24:13.138545 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:13.140698 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:13.155787 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 45453us
W20240502 08:24:13.246829 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.333032 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:13.339571 21616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 55083.6us
W20240502 08:24:13.372875 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:13.384969 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.439000 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.459759 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.544595 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.556838 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:13.610090 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:13.666404 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:13.683862 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:13.706362 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:13.752312 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:13.773859 21485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 90316.2us
I20240502 08:24:13.784691 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:13.795820 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:13.843462 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:13.901760 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:13.919911 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:24:13.928246 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:13.971081 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:13.989975 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:13.999938 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:14.045239 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:14.089910 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:14.131914 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:14.140864 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:14.152428 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:14.188961 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:14.189436 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:14.231278 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:14.240445 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:14.242722 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:14.248545 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:14.298099 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:14.422596 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:14.448405 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:14.457578 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:14.482296 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:14.488584 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:14.493405 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:14.510190 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:14.514848 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:14.571625 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:14.576046 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:14.618887 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:24:14.642927 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:14.659199 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:14.703143 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:14.742687 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:14.745990 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:14.753029 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:14.775126 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:14.847186 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:14.850567 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 08:24:15.045169 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:15.153322 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:15.177989 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:15.189508 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:24:15.190598 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:15.197304 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:15.218830 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:15.220211 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:15.228991 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:15.240001 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.251932 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.306905 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:15.312345 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.368927 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.374907 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:15.382006 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:15.458225 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:15.468956 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:15.476898 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:15.482127 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:15.492451 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:15.524727 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:15.578997 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:15.580680 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:15.581032 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:15.592069 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:15.629393 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.692807 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:15.746863 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:24:15.787021 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:15.913234 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:15.937530 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:15.954797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:15.971269 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:15.989756 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:16.016451 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:16.066488 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:16.077294 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:16.100843 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.110772 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:16.130380 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:16.131872 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:16.141292 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:16.144440 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.159097 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:16.169462 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:16.172420 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:16.212570 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.216890 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:16.229305 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:16.248766 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.267750 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:16.302695 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:16.313094 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:16.313467 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.317725 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:16.379380 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 51753.2us
W20240502 08:24:16.388453 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:16.399053 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:16.399349 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:16.486726 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:16.712235 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:16.748368 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:16.800505 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:16.841807 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:16.857125 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:16.878221 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:16.889628 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:16.891321 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:16.945534 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:24:16.991370 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:17.055063 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:17.078768 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:17.100131 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:17.100786 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:17.103001 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:17.104750 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:24:17.117645 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:17.118786 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:24:17.141345 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:17.145180 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:17.283174 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:17.291601 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:17.369118 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:17.417702 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:17.424968 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:17.447724 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:17.462075 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:17.469138 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:17.512207 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:17.694958 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:17.725909 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:17.774228 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:17.828724 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:17.861651 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:17.864584 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:17.864686 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:17.870656 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:17.876081 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:17.897424 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:17.904891 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:17.911720 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:17.921715 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:17.923223 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:17.942330 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:24:17.944336 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:17.975750 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:17.979048 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:17.979290 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:18.001905 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:18.059336 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:18.077668 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:18.099539 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:18.116290 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.121327 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.124610 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:18.131927 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:18.132949 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:24:18.161994 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:18.167091 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:18.175827 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:18.206023 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:18.209637 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:18.215427 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:18.236790 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.393810 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:18.414405 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:18.483644 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:18.491791 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:18.497720 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:18.500159 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:18.544296 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:18.547406 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:18.595826 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.601922 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.673897 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.783387 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.797401 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:18.833906 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:18.882926 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:18.897365 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:18.919178 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:18.921805 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:24:18.934340 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:18.938483 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:18.942286 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:18.944032 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:18.954718 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:19.089236 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:19.094696 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:19.288213 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:19.309646 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:19.313354 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:19.316452 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:19.319948 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:19.323367 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:19.346360 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:19.369657 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:19.383635 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:19.430204 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:19.504563 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:19.504652 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:19.548511 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:19.576920 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:19.594151 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:19.652935 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:19.713264 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:19.714313 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:19.720152 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:19.724275 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:19.744000 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:19.748080 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:19.773380 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:19.801272 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:24:19.809422 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:19.838224 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:19.943818 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:24:19.996657 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:20.043624 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:24:20.113340 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:20.124749 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.133112 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:20.134253 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.157481 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:20.167639 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:20.171422 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.197245 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 08:24:20.222739 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.227205 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:20.246798 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.254333 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:20.386811 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.399700 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:20.402761 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:20.403015 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:20.415094 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:20.423538 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:20.423962 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.435443 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:20.527010 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.552099 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:20.606509 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.620133 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.627157 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:20.635994 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:20.640125 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:20.681694 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:20.776810 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:20.910318 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.933899 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:20.948051 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:20.953557 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:20.957020 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:20.959578 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:20.994630 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.012290 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:21.017434 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.025851 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:21.058832 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.059712 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:21.067440 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:21.076315 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 111678us
I20240502 08:24:21.109000 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:21.119534 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:21.125356 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.131598 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.136864 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:21.146906 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:21.151793 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:21.181598 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:21.223191 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:21.254253 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:21.329953 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:21.497798 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:21.522886 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.523988 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.556753 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:21.562464 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:21.603056 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:21.675622 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.697018 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.712596 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:21.727525 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.747851 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:21.764648 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:21.780282 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:21.793148 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:21.804921 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:21.820523 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:21.833261 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:21.849319 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 117341us
W20240502 08:24:21.869251 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:21.889578 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.902289 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:21.902315 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:21.927451 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:21.937788 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:24:21.937983 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:21.955567 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:21.956589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:22.013732 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:22.066659 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:22.072311 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:24:22.185351 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:22.316043 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:22.370954 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:22.501444 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:24:22.556907 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:22.563331 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:22.588373 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:22.595489 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:22.613729 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:22.624408 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:22.722509 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:22.781855 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:22.796607 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:22.813319 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:22.844981 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:22.870014 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:22.872532 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:22.913909 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:22.914923 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:22.925760 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:22.933411 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:22.934198 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:22.943521 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:22.955896 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:22.999609 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:24:23.001138 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.044588 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:23.052403 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:23.110911 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:23.121022 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.131120 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:23.176195 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:23.179168 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:23.204711 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:23.211982 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:24:23.219925 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.227295 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:23.258553 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:23.274305 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:23.299316 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:23.314347 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:23.321590 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.324827 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:23.331763 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:23.341346 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:23.367831 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:23.372757 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:23.497315 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:23.597880 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.600000 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:23.605402 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:23.606743 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:23.608147 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:23.648586 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:23.688217 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:23.714213 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:23.767913 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:23.817641 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:23.938403 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:24.047355 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:24.102780 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:24.173094 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:24.193953 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:24.213930 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:24.220245 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:24.226254 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 48798us
W20240502 08:24:24.235018 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:24.244599 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:24.278198 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:24.278368 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:24.301532 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:24.328960 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:24.359833 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:24.361517 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:24:24.369143 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:24.422765 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:24:24.432319 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:24.460350 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:24.489154 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:24.529526 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:24.538144 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:24.558423 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:24.568518 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:24.606299 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:24.612553 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:24.647202 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:24.652565 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:24.698053 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:24.698494 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:24.728068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:24.730437 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:24.766750 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:24.770956 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:24.772552 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:24.830276 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:24.839535 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:24.855661 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:24.973596 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:24.988543 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:24:24.997938 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:25.032665 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:25.138859 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:25.152104 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:25.152575 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:25.164470 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:25.198262 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:25.218746 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:24:25.219409 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:25.221055 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:25.226612 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:25.260500 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 89683.1us
W20240502 08:24:25.331573 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:25.352901 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:25.452105 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:24:25.540419 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:25.559736 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:25.587868 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:25.588145 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:25.642400 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:25.682174 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:25.714856 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:25.725371 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:25.766260 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:25.774282 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:25.808161 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:25.824762 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:25.833277 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:25.845014 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:25.856612 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:25.875077 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:25.916425 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:25.948585 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:25.964308 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:25.979486 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:25.996213 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.003538 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:26.008277 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:26.031986 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:26.048810 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:26.068392 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:26.120095 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:26.134629 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 118790us
I20240502 08:24:26.138653 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:26.143347 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:26.167151 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:26.173259 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:26.190196 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:26.192323 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.193677 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:26.194073 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:26.227515 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.233263 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:26.317890 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:26.327232 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:26.387133 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:26.562844 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.625324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:26.651046 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:26.651494 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:24:26.667776 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:26.702013 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:26.705327 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:26.720841 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:26.737910 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:26.741303 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:26.768214 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:26.773579 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:26.782238 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.798280 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:26.801021 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:26.831329 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:26.965283 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.081248 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:27.163766 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:27.172067 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:27.176563 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:27.177475 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:27.209702 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.277228 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:27.294392 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.305466 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:27.307148 23407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 52611.7us
W20240502 08:24:27.332975 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:27.351421 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:27.360364 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:24:27.374878 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:27.404908 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.425333 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:27.427807 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:27.453547 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:27.461249 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:27.495671 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:27.495889 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:27.506870 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:27.524806 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:27.557300 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:27.568074 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:27.571550 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.593986 21924 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:27.603755 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:27.606720 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:24:27.614776 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:27.631650 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:27.700788 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:27.734889 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:27.825369 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:27.846993 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:27.908416 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:27.963749 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:27.990473 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:28.016106 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:28.018792 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:24:28.123374 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:28.165163 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:28.181423 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:28.202986 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:28.205668 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:28.211169 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:28.237437 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:28.247682 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:28.253389 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:28.298826 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:28.334537 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:28.339709 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:28.340714 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:28.402716 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:28.413936 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:28.454139 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:28.482771 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:28.486385 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:28.522097 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:24:28.546576 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:28.600463 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:28.652478 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:28.827992 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:28.852728 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:28.856884 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:28.877521 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:28.885388 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:28.885787 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:28.890667 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:28.925444 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:28.940553 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:28.960058 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:28.963169 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:24:28.970624 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.003473 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.053169 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:29.120536 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.126402 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:29.155325 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:29.163506 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:29.171869 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:29.173252 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:29.188439 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.227499 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:29.244112 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:29.259120 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:29.289687 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:29.312319 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:29.317768 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:29.355942 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:29.398804 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:29.418980 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:29.518265 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:29.532943 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:29.597926 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.618736 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:29.624228 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:29.642517 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.659587 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:29.690898 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:29.761070 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:24:29.762307 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:29.797787 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:29.880402 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:29.894598 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.951023 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:29.959597 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:29.960851 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:29.975050 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:29.977522 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:30.003619 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:24:30.008127 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:30.023562 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:30.045109 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:30.119764 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:30.120435 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:30.129036 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:30.178449 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:30.182454 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:30.229363 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:30.246433 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.308825 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 08:24:30.309651 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.320343 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:30.382875 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:30.389199 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.397755 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.485108 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:30.509395 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.597971 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:30.657456 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:30.664209 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:30.696270 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:30.704411 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:30.731943 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:30.738680 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:30.760823 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 55956.8us
I20240502 08:24:30.785719 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:30.838116 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:24:30.877959 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:30.886214 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:30.972281 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:31.016026 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:31.087750 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:31.121838 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:31.133227 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:31.139725 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:31.159631 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:31.161347 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:31.165102 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:31.168749 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:31.183473 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:31.195369 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:31.203214 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:31.211882 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:31.229701 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:31.276003 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:31.295766 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:31.301292 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:24:31.343992 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:31.384692 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:31.388064 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:31.389989 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:31.434722 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:31.437227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:31.552305 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:31.560997 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:31.580425 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:31.585229 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:31.619347 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:31.646522 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:31.722076 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:31.812500 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:31.908298 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:31.909495 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:31.980167 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:32.100270 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:32.103547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:32.108139 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:32.146793 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:32.193452 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:32.218869 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:32.220134 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:32.220603 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:32.235500 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:32.252298 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:32.385952 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 94814.9us
W20240502 08:24:32.422060 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:32.451042 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:24:32.464785 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:32.476085 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:32.484225 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:32.515723 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:32.527172 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:32.566437 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:32.592193 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:32.616938 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:32.618695 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:24:32.640640 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:32.671557 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:32.760155 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:32.779866 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:32.809128 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:32.814771 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:32.826432 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:32.832368 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:32.844292 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:32.869925 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:32.924690 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:32.942297 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:32.958050 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:32.965343 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:32.971797 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:32.992377 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:32.995733 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.024978 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:24:33.025135 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:33.025633 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:33.035902 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.037880 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:24:33.044076 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:33.054185 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.062076 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:33.299068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:33.361557 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:33.387336 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:33.454281 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:33.461941 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:33.475100 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:33.521374 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.576745 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:33.604233 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:33.608717 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.638731 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.677551 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:33.718046 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:33.819900 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:33.873898 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:33.891008 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:33.903501 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:33.923506 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:33.934571 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:33.973311 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:24:34.039530 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:34.077381 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:34.139726 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:34.153816 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:34.166033 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:34.174806 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:34.179322 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:34.181927 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:34.184017 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:34.212843 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:34.232035 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:34.241025 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:34.268239 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:34.294329 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:34.305006 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:34.324960 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:34.325425 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:34.338574 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:34.393317 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:34.399955 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:34.463227 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:34.514343 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:34.543349 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:34.558552 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:34.581075 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:34.601692 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:34.625339 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:34.634465 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 46766.5us
I20240502 08:24:34.644675 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:34.681710 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:34.724607 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:34.732175 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:34.762209 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:34.782200 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:34.817431 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:24:34.854831 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:34.866596 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:34.889251 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:34.898070 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:34.898581 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:35.063879 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:24:35.096287 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:35.119781 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:35.148741 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:35.202252 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:35.222712 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:35.231400 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:35.243757 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:35.248131 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:35.270977 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:35.274299 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:35.307482 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:35.336922 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:35.337878 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 08:24:35.382138 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:35.410019 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:24:35.461395 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:35.482810 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:35.525135 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 79368.2us
W20240502 08:24:35.527213 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:35.547660 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:35.550364 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:35.557178 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:35.560748 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:35.598809 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:35.598784 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:35.625532 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:35.669155 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:35.745791 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:35.747618 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:35.899434 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:35.999691 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:36.034128 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:36.037149 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:36.049698 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:36.077119 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:36.085335 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:36.086467 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:36.095378 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:36.111557 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:36.120924 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:36.121598 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:36.127913 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:36.149976 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:36.202100 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.246955 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.259984 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:36.330235 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:36.393150 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:36.417194 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:36.423311 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:36.465705 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:36.472950 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:36.474637 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:36.528501 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:36.537326 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:36.594144 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:24:36.655871 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:36.670044 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.757153 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.775803 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.785125 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.837764 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:36.834131 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:36.857015 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:36.920322 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:36.987057 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:37.033578 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:37.057739 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:37.072847 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:37.073665 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:37.104241 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:37.122107 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:37.138672 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:37.159000 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:37.160250 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:37.198428 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:37.205850 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:37.223176 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:37.234440 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:37.267222 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:37.290055 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:37.303297 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:37.322818 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:37.334939 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:37.344514 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:37.405472 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:37.444201 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:37.451851 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:37.486508 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:37.575598 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:37.588503 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:37.639868 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:24:37.641309 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:37.649605 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:37.671540 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:37.711464 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:37.725911 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:37.779671 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:37.798470 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:37.842737 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:37.856575 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:37.883816 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:24:37.884806 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:37.906348 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:37.934162 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:37.936172 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:37.980907 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:37.996440 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:38.005954 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:38.011929 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:38.028882 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:24:38.045289 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:38.069382 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:38.134084 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:24:38.271972 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:38.364025 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:38.391837 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:38.403599 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:38.410240 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:38.412878 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:38.416039 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:38.454252 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:38.518316 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:38.550365 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:38.652706 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:38.657459 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:38.662925 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:38.667644 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:38.720187 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:38.726403 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:38.735903 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:38.743930 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:38.778048 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:38.817020 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:38.831398 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:38.852551 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:38.874888 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:38.911778 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:38.915782 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:38.930385 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:38.931306 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:38.968468 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:39.054769 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:39.139062 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:39.147550 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:39.152808 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:39.167924 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:39.196111 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:39.308658 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:39.308696 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:39.332962 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:39.351696 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:39.355033 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:39.372175 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:39.381060 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:39.381620 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:24:39.405174 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:39.420205 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:39.453683 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:39.534104 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:39.556378 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:39.560501 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:39.566524 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:39.578902 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:39.612849 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 65206.4us
I20240502 08:24:39.628983 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:39.716547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:39.748095 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:39.757313 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:39.762660 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:39.801262 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:39.804461 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:39.806480 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:24:39.833137 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:39.838202 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:39.936053 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:39.991891 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:40.010344 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:40.032706 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:40.043960 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:40.055333 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:40.074294 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:40.092692 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:24:40.095536 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 54739.4us
W20240502 08:24:40.116273 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.127965 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.235641 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:24:40.242571 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.260241 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:40.271795 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:40.327988 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:40.354979 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.362325 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:40.367074 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:40.372561 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:40.372650 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:40.384366 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:40.412026 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:40.412684 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 08:24:40.413810 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 68430.1us
W20240502 08:24:40.413964 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:40.574398 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.619531 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:40.623530 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:40.629133 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.640138 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:40.714038 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:40.734614 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:40.825841 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:40.863839 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:40.871133 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:40.914290 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:40.929327 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:40.992656 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:41.025728 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.026947 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:41.042040 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:41.049805 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.060850 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:41.074931 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:41.103789 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:41.107587 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:41.120198 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:41.183997 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:41.389619 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.398447 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:41.427722 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:41.446211 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:41.482684 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:41.482822 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:41.486095 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.496608 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:41.509162 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:41.512800 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:41.513208 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:41.551682 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:41.571317 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.599148 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:41.628978 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:41.642714 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:41.668848 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:41.693827 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:41.715659 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:41.731149 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:41.754606 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:41.754619 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:41.916728 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:24:41.917362 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:41.924851 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:41.953979 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:42.024539 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:42.080536 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.112948 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:42.136591 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:42.138921 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:42.188966 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:42.208608 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:42.237891 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:42.305217 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:42.335191 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:42.478672 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.494282 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.522078 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:42.556932 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:42.563441 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:42.580680 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:42.584129 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:42.584616 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:42.586844 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:42.600265 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:42.605348 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:42.619009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.658489 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:42.703457 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:42.729773 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:42.833300 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:42.848052 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:42.862043 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:42.927564 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:42.958259 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:24:42.960474 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.975812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:42.978328 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:42.980309 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:42.997974 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:43.012459 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:43.015771 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:43.042470 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:43.060081 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:43.088578 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:24:43.131461 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:24:43.136021 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:43.142436 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:43.146277 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:24:43.157613 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:43.166540 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:43.170034 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:43.216909 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:43.248822 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:43.277520 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:43.287869 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:43.313009 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:43.317281 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:43.470686 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:43.490942 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:43.717654 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:43.838177 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 120519us
W20240502 08:24:43.888530 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:43.904142 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:43.930256 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:43.978324 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:43.992413 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:44.017531 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:44.042536 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:44.071903 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:44.083634 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:44.212739 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:44.214458 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.229822 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:44.236845 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:44.240424 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:44.246356 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:44.249680 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:44.255211 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:44.255530 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:44.273244 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:44.305122 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:44.314100 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:44.323045 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:44.339006 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:44.345152 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:44.473892 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:44.487272 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:44.487797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.499090 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:44.509567 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.539309 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:44.543906 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:44.556650 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:24:44.557464 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:44.567974 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:44.573226 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:44.606981 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:24:44.619048 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.662050 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:44.715502 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:44.742233 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:44.818684 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:44.837947 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.857107 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:44.884784 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:44.913930 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:44.922801 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:44.948719 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:45.020206 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.048779 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:45.129412 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:24:45.130901 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.253213 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:45.341181 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.345183 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:45.359687 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:45.420743 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:24:45.461386 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:24:45.522181 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.534222 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.620307 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:45.637341 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.647030 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:45.661338 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.663928 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:45.708040 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:45.725237 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:45.865978 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:45.867170 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:45.870195 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:45.902119 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:45.927433 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:45.928951 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:45.943806 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:45.971864 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:45.972931 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:45.973649 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:45.980926 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:45.988770 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:46.010586 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.015198 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:46.072819 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:46.116468 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:46.120733 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:46.134308 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.145887 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:46.155035 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:46.206015 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:46.229840 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:46.240483 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.287067 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:46.287321 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:46.302857 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 77449.8us
I20240502 08:24:46.306501 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:46.315788 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.332758 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:46.360781 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:46.456867 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:46.584419 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:46.639806 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.661018 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:46.679633 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:46.710112 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:46.716365 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:46.757040 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.762099 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:46.780539 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:46.781893 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:46.795683 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:46.809743 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:46.832567 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:46.839934 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:46.868587 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.871480 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:46.891450 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:46.917141 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:46.922633 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:46.932240 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:46.967542 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:46.980202 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:47.060720 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:47.160743 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:47.170446 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:24:47.203023 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:47.393150 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:47.405010 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:47.479809 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:47.493976 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:47.529652 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:47.549103 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:47.555495 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:47.555472 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:47.564404 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:47.594455 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:47.635636 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:47.637658 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:47.639720 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:47.640460 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:47.675120 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:47.695461 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:47.704715 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:47.705492 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:47.707355 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:47.724630 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:47.752763 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:47.753634 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:47.835932 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:47.861567 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:47.880869 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:47.901855 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:47.944118 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:48.017676 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:24:48.020550 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:48.073583 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:48.099807 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:48.122990 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:48.129601 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:48.137390 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:48.160331 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:48.173107 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:48.175621 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:48.180079 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:48.223349 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:48.237195 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:48.260689 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:24:48.284794 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:48.305498 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:24:48.350401 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:48.373669 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:24:48.394094 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:48.557003 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:48.558193 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:48.566226 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:48.579641 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:48.599546 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:48.630362 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:48.631510 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:48.656497 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:48.738492 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:48.930030 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:48.930894 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:48.971714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:49.002287 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:49.004513 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:49.014278 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:49.030289 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:49.030794 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:49.043885 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:49.068608 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 142804us
W20240502 08:24:49.073477 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:49.108750 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:49.129808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:49.145162 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:49.208966 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:49.258579 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:49.283732 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:49.287678 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:49.289193 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:49.296259 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:49.310225 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:49.325803 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:49.326331 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:49.332839 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:49.348901 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:49.406410 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:49.428215 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:49.461246 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:49.465785 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:49.466576 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:49.477430 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:49.491922 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:49.503683 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:49.542660 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:49.550768 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:49.551179 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:49.612289 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:49.678072 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:49.724910 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:49.847131 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:49.900839 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:24:49.966836 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:49.984038 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:49.996809 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:50.025367 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:50.031031 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:50.043967 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:50.084275 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:50.109826 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:50.128859 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:50.188825 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:24:50.190284 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:50.203316 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:50.213474 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:50.223218 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:50.230013 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:50.258668 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:50.292253 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:50.310830 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:50.325796 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:50.381520 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:50.440191 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:50.473300 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:24:50.475533 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:24:50.495517 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:50.555693 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:50.575130 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:24:50.576804 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:50.605621 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:50.746385 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:50.785416 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:50.789709 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:50.797299 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:50.873196 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:50.894991 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:50.897020 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:50.905148 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:50.910126 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:50.917276 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:50.939745 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:50.951385 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:50.986688 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:50.993741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:51.017894 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:51.019202 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:51.039649 21747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:41299, user_credentials={real_user=slave}} blocked reactor thread for 44822.4us
I20240502 08:24:51.039767 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:51.041234 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:51.048468 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:51.073107 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:51.073959 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:51.086513 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:51.146759 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:51.191619 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:51.201321 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:51.392546 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:51.479987 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:51.552654 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:51.558126 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:51.568562 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:51.576808 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:51.603844 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:51.630208 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:51.654552 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:51.680557 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:51.710222 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:51.713850 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:51.737977 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:51.756768 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:51.776875 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:51.781013 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 94511us
I20240502 08:24:51.784643 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:51.787009 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:51.793126 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:51.832424 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:51.862602 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:51.927335 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:51.945358 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:52.095719 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:52.097478 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:52.163007 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:52.164536 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:52.172219 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:52.172884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:52.200667 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:52.211328 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:52.230646 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:52.252753 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:52.253010 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:52.301874 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:52.325035 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 48627.2us
W20240502 08:24:52.414698 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:52.432178 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:52.484640 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:52.534009 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:52.550086 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:52.555608 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:52.559054 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:52.581979 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:52.595291 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:52.600710 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:24:52.626452 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:52.635553 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:52.651360 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:52.700403 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:52.704182 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:52.704841 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:52.723171 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:52.797364 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:52.798048 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:52.827831 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:52.925884 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:52.955915 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:53.072196 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:53.088800 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:53.133209 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:24:53.135150 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:53.197276 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:53.203583 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:53.267879 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:53.287828 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:53.310424 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:53.321678 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:53.343449 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:53.354368 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:53.364416 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:53.382064 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:53.384837 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:53.406255 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:53.418164 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:53.422506 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:24:53.423645 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:24:53.444067 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:53.446022 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:53.458443 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:53.465045 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:53.468523 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:53.484912 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:53.512773 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:53.541302 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:53.562603 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:53.565768 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:24:53.633589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:53.665657 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:53.718672 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:53.752844 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.000243 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:54.000583 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.054272 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:54.116753 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:54.173795 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.222507 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 67288.4us
I20240502 08:24:54.250572 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:54.251052 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:54.277648 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:54.305230 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:54.315883 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:54.324538 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:54.324901 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:54.346783 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:54.350481 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:54.356158 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:54.361143 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:54.404568 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:54.416764 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:54.441875 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:54.493650 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:54.560614 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:54.571882 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:54.601521 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.658793 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:54.732324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:54.745966 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.756705 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:54.789999 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:54.794165 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:54.799167 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:54.820015 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:54.819972 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:54.873569 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:54.875697 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:54.941032 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:54.969125 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:54.983745 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:55.008560 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:55.015636 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:24:55.027376 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:55.044591 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:55.045773 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:55.059271 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:55.061590 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:55.072245 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:55.081143 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:55.100435 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:55.101598 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:55.135298 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:55.140041 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:55.250309 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:24:55.282059 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:55.324455 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:55.334224 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:55.424535 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:55.524962 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:55.541855 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:55.557226 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:55.577036 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 08:24:55.590739 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:55.620432 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 08:24:55.621591 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:55.656610 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:55.668152 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:55.706722 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:55.723342 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:55.726248 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:55.733847 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:55.746304 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:55.750041 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:24:55.822567 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:55.828218 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:55.834082 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:55.853477 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:55.853876 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:55.868228 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:55.886209 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:55.937801 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:55.947815 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:55.977412 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:55.988039 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:55.988147 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:56.001968 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:56.046092 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:56.065474 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:56.200451 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:56.246806 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:56.276264 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:56.299223 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:56.393447 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:56.394158 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:24:56.434598 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:56.475780 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:56.514621 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:56.517961 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:56.547453 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:56.554621 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:56.594528 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:56.607542 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:56.622889 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:56.647130 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:56.652045 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:56.670701 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:56.672742 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:56.688668 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:56.946646 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:56.977291 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:57.052439 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.057384 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:57.076051 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.177683 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:57.212704 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.217218 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.242671 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.245476 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.287164 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:24:57.321516 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:57.324983 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.325003 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:57.348987 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:57.394311 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.399648 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:57.457109 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:57.484695 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:24:57.503912 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:57.558722 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.577872 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:57.578352 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.592716 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:57.628062 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:57.638597 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.650374 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:57.658879 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.663825 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:57.682315 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:57.697044 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:57.700785 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:57.735913 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:57.740339 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.755139 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:57.766191 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:57.777626 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:57.783149 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:57.801688 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:57.848111 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.851161 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:57.937124 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:57.965400 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:58.055061 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:58.074501 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:58.078567 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:58.097957 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:58.110971 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:58.140497 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:58.148741 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:58.164937 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:24:58.166433 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:58.184307 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:58.260986 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:58.267793 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:58.345635 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:58.358819 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:24:58.388464 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:58.422237 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:58.467779 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:24:58.507187 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:58.525154 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:58.553823 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:58.629832 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:58.664394 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:58.674423 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:58.683377 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:58.764256 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:24:58.857333 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:24:58.909198 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:58.929672 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:58.999766 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:59.025638 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:59.027709 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:24:59.027981 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:59.055402 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.073426 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:59.083599 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:59.096904 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:59.118367 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:24:59.167042 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.214550 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:24:59.301880 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:59.313922 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:24:59.322979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:59.330534 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:59.356680 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:59.381096 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:59.435889 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:24:59.456724 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:24:59.459606 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:24:59.463635 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:24:59.466638 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.473130 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:24:59.503173 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:24:59.531373 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:59.551421 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.598510 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:24:59.610241 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:24:59.634353 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:24:59.689589 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:24:59.726126 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.745919 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.790488 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:24:59.811471 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:24:59.872659 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:24:59.958647 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:24:59.975288 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:00.020726 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:00.028625 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:00.031440 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.042018 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:00.069438 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:00.084573 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:00.107141 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:00.132664 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:00.152874 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:00.202013 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:00.218669 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.219295 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:00.276106 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:00.287791 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:00.317375 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:25:00.318594 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:25:00.517458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:00.524502 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:00.532706 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:00.556061 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:00.593120 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:00.609645 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:00.624708 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 08:25:00.638793 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:00.639405 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:25:00.676630 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:00.694550 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:25:00.794776 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:00.798852 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:25:00.808104 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:25:00.838245 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:25:00.869760 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:00.873479 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:00.883237 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.896540 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:00.915625 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:00.917366 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:00.928517 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:00.943034 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.968760 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.989681 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:00.989933 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:00.994140 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:01.023684 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:01.090822 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:25:01.108259 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:01.126907 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:01.146247 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:01.242187 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:01.253705 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:01.263063 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:01.265482 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:01.277846 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:01.346707 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:01.370550 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:01.389597 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:01.401146 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:01.522914 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:01.529110 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:01.553876 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:01.574726 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:01.697319 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:01.712013 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:01.932626 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:01.940562 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:01.952380 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:01.988775 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:01.996544 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:02.015846 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:02.020134 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:02.020291 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:02.073868 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.095016 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:02.107160 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:02.108392 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:02.124246 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:02.133600 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:02.142298 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:02.162897 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.168898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.177033 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:02.180300 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:02.197311 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:02.219594 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:25:02.227066 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:02.246140 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:02.253742 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:02.271126 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:02.299847 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:02.309904 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:02.334055 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:02.371953 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:02.485198 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:02.495291 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:02.514415 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:02.522539 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.541743 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:02.548342 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:02.565198 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:02.580947 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:02.587631 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.763823 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:02.770725 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:02.843266 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:02.860191 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:02.875567 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:02.877213 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:02.891590 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:02.970153 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:02.984951 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:02.986186 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:03.016692 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:03.129458 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:03.197441 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:25:03.198591 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:03.201766 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:25:03.285431 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:03.292121 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:03.314509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:03.316570 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:03.331858 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:03.355852 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:03.366225 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:03.370044 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:03.398578 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:03.399009 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:03.448412 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:03.453442 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:03.468884 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:03.481060 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:03.505373 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:03.511534 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:25:03.564764 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:03.686300 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:03.713043 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:03.716881 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:03.725016 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:03.731544 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:03.752775 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:03.780673 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:03.812654 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:03.828862 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:03.832598 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:03.842249 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:03.845222 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:03.865546 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:03.889523 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:03.900650 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:03.910028 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:03.910693 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:03.922050 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:03.927491 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:03.928913 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:04.095914 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:04.156575 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:04.268271 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:04.275936 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:04.292991 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:04.308472 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 59971.2us
I20240502 08:25:04.310664 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:04.327091 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:04.346751 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:04.362892 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:04.367981 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:04.379573 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:04.405170 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:25:04.406622 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:04.418022 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:04.469976 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:04.631000 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:25:04.634075 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:04.648103 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:04.681887 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:04.690402 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:04.690613 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:04.753544 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:04.782605 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:04.804842 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:04.837653 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:04.877382 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:04.878634 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:04.890197 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:04.984870 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:05.067492 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:05.084559 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:05.111014 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:05.121592 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:05.160521 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:05.178222 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:05.187072 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:05.295092 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:05.315119 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:05.321913 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 08:25:05.336813 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:05.339149 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:05.363612 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:05.364567 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:05.393175 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:05.397646 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:05.485304 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:05.578775 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:05.596550 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:05.613174 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:25:05.614094 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:05.649243 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:05.657742 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:05.658876 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:05.659752 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:05.663682 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:05.666328 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 08:25:05.676971 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:05.680025 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:05.687546 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:05.697409 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:05.711884 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:05.738700 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:05.761955 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:05.772541 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:05.836383 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:05.870826 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:05.887959 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:05.922191 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:05.960618 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:06.129606 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:06.179935 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:06.189863 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:06.194386 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:06.235605 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:06.243733 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:25:06.244762 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:06.300487 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:06.308449 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.320600 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:06.321136 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:06.382926 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:06.438505 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:06.560884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.584504 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:06.608631 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:06.620281 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.621078 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:06.622463 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:06.636384 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:06.677354 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:06.687129 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:06.804033 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:06.813355 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.835315 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:06.852932 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:06.853389 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:06.862742 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:06.871081 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.901278 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:06.913961 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:06.916672 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:06.952700 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:06.973364 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:06.998239 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:07.007877 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:07.011583 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:07.053978 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:25:07.063892 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:07.076934 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:07.112179 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:07.218408 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:07.277649 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:07.307628 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:07.336367 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:07.386068 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:07.393893 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:07.400019 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:07.414796 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:07.433488 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:07.463939 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:07.492461 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:07.499972 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:07.574117 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:07.607618 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:07.734187 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:07.740864 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:07.757846 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:07.779924 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:07.815678 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:07.840864 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:07.846431 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:07.874475 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:07.897891 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:07.993520 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:08.057124 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:08.061305 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:08.167503 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:08.242071 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:08.258818 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:25:08.260501 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:08.278879 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:08.305792 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:08.307092 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:08.314841 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:08.325886 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:08.381815 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:08.391594 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:08.417551 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:25:08.430145 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:08.512472 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:08.537905 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:08.580453 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:08.645829 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:08.755775 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:08.778334 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:08.789192 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 66139.7us
I20240502 08:25:08.803907 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:08.812369 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:08.819169 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:08.822052 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:25:08.854658 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:08.889480 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:08.889698 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:08.901965 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:08.954329 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:08.975354 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:08.986089 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:09.019755 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:09.072038 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:09.076002 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:09.117224 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:09.123268 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:09.231613 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:09.240770 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:09.244258 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:09.278630 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:09.314770 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:09.326038 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:09.333722 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:09.355206 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:09.373176 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:09.374933 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:09.411199 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:09.417505 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:09.530151 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:09.549219 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:09.560510 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:09.576509 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:09.613039 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:09.638648 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:09.660799 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:25:09.676221 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:09.770795 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:09.814826 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:09.818496 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:09.877372 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:25:09.878584 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:09.888834 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:09.992017 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:10.098150 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:10.131196 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:10.193450 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:10.201016 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:10.222337 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:10.243703 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:10.246657 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:10.260244 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:10.281258 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:10.295665 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:10.325789 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:10.327023 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:25:10.412504 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:10.464742 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:10.490180 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:10.491710 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:10.514869 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:10.537097 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:10.539544 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:10.541180 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:10.542825 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:10.553124 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:10.695864 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:25:10.703158 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:10.733508 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:10.767530 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:10.794667 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:10.795274 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:10.807511 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:10.815922 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:10.861660 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:10.862329 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:10.869827 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:10.882438 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:10.887420 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:10.899474 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:10.906948 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:10.911995 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:10.948634 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:25:10.998283 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:11.080842 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:11.097851 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:11.105526 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:11.109990 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:11.130002 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:11.169490 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 51806.5us
I20240502 08:25:11.171437 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:11.254249 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.265537 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:11.284142 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:25:11.287359 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:11.331375 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 51829.8us
I20240502 08:25:11.368928 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:11.441622 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:11.484019 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:11.484213 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:11.502517 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:11.527591 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.529822 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:11.545130 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:11.547001 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.559476 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:11.580924 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:11.581552 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:11.611377 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.716342 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:11.743767 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:11.787328 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:11.879851 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:11.900074 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.902029 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:11.974581 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 71430.7us
W20240502 08:25:11.978229 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:11.979967 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 75699.1us
I20240502 08:25:11.983342 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:12.003592 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:12.051781 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:12.068081 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:12.098322 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:12.108677 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:12.140355 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:25:12.238257 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:12.261214 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:12.346035 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:12.399350 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:12.413904 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:12.467995 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:12.493412 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:12.493992 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:12.571133 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:12.583909 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:12.585028 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:12.662930 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:12.685089 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:12.690879 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:12.716472 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:12.718932 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:12.744391 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:12.767930 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 46252.8us
I20240502 08:25:12.779356 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:12.795439 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:12.795626 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:12.815054 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:12.822971 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:12.829996 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:12.887383 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:13.049438 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:13.072964 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:13.105795 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:13.132486 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:13.136301 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:13.155764 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:13.158888 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:13.166015 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:13.169206 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:13.177686 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:13.183607 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:13.186394 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:13.198836 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:13.245095 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:13.261435 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:13.281642 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:13.313613 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:25:13.500281 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.517300 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:13.554183 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:25:13.614213 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:13.673739 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:13.676607 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:13.686023 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:13.690042 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:13.707841 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:13.719499 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:13.720867 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.723896 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.801676 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:13.889657 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.914628 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.928314 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:13.962875 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:25:13.989401 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:14.133879 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:14.192236 23407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 66580.9us
I20240502 08:25:14.192966 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:14.193923 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:14.200194 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:14.211954 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:14.253791 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.272488 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:14.299958 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:14.319645 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:14.341184 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:14.364372 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:14.401913 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:14.415078 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:14.423800 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:14.432271 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.440028 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:14.443641 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:14.455161 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.457883 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:14.465970 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.492357 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:14.504830 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:14.526109 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:14.608520 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:14.624663 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:14.638725 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:14.682372 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:14.686470 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:25:14.706071 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:14.714866 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:14.748440 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:14.748797 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:14.804584 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:14.863687 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.893641 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:14.902789 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:14.911729 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:14.960353 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:14.979713 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:15.125553 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:25:15.126865 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:15.150195 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:15.174826 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:15.182428 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:15.184806 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:15.205981 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:15.229460 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:15.243723 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:15.273598 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:15.285204 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:15.301012 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:15.340204 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:15.375224 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:25:15.378511 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:15.378760 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:15.438293 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 46483.2us
I20240502 08:25:15.639214 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:15.640710 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:15.684077 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:15.694351 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:15.699424 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:15.702940 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 08:25:15.704725 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:15.705530 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:15.747315 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:15.758929 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:15.791481 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:15.808651 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:15.836122 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:15.890084 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:15.923940 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:15.948601 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:15.960140 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:15.991871 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:16.047983 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:16.051585 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:16.074003 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:16.077939 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:16.104104 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:16.121387 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:16.193660 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:16.214525 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:16.230273 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:25:16.288640 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:25:16.290068 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:16.304359 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:16.339700 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:16.342243 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:16.372923 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:16.373070 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:16.422211 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:16.440320 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.474970 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.475292 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.511852 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.542500 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.595036 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:16.608548 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:16.636718 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:16.656150 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:16.663669 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:16.664417 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:16.672556 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:16.689206 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:16.694504 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:16.732754 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:16.746292 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:16.786065 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:17.003844 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.120241 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:17.123410 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.163766 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:17.172581 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:17.177902 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:17.235440 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:17.239881 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:17.244058 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:17.258913 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:17.273051 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:17.351744 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:17.364005 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:25:17.367617 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:17.382298 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:17.390429 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:17.404340 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:17.405584 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:17.425603 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:17.444499 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:17.524951 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:17.553357 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:17.637686 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:17.666564 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:17.685640 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:17.709107 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.719455 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:17.733678 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.789636 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:17.789834 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:17.797442 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:17.840011 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:17.851613 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:17.874975 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.905465 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:17.911547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:17.916060 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:17.941161 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:17.971117 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:17.975010 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:17.980186 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:18.074996 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:18.088681 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:18.219877 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:18.271721 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:18.317428 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:18.324527 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:18.370208 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:18.380539 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:25:18.381574 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:18.390889 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:18.408358 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:18.431872 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:18.434085 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:18.518862 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:18.534716 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:18.577797 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:18.587973 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:25:18.621865 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:18.635339 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:18.639206 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:18.667446 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:18.670274 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:18.699329 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:18.704859 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:18.725589 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:18.755873 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:18.817313 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:18.835939 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:18.837215 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:18.840493 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:18.881626 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.035108 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:25:19.039925 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:19.097752 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.099224 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.112813 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.118721 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.212213 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.248704 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:19.349566 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:19.368693 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:19.407004 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:19.408850 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:19.438012 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:19.444363 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:19.471001 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:19.497640 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:19.506623 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:19.518617 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:19.573186 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:19.579013 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:19.607758 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:19.634899 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:25:19.648300 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:19.656293 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:19.666805 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:19.672183 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:19.754333 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:19.914732 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:19.920012 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:19.936587 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:19.970305 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:19.975740 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:19.976696 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:20.007681 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:20.056687 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:20.117833 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:20.143889 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:20.165459 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:20.184540 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:20.215618 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:20.219868 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:20.221124 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:20.273576 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:20.297748 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:20.315519 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:20.344619 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:20.352533 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:20.402117 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:25:20.403312 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:20.428627 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:25:20.430086 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:20.451486 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 44075.8us
W20240502 08:25:20.470408 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:20.473497 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:25:20.553472 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:20.589249 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:20.596362 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:20.611176 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:20.620434 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:20.659005 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:20.711839 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:20.718852 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:20.727806 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 08:25:20.787933 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:20.919776 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:20.971917 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.010753 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:21.016319 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.064817 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:21.067525 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:21.101907 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:21.122653 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.142844 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:21.163830 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:21.198318 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:21.202970 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:21.218966 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:21.232173 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:21.241741 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:21.273977 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:21.278585 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.330089 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.340473 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:25:21.358850 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.359824 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:21.439508 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.534687 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.549855 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.599689 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:21.601131 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.616467 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.678114 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:21.685081 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:21.694433 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:21.709283 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:21.709802 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:21.711310 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:21.712406 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:21.718927 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:21.730818 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:21.773962 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.791427 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:21.792023 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:21.808252 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:21.815382 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:21.836556 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:21.879236 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:22.019193 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:22.208535 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:22.217828 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:22.234054 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:25:22.324048 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:22.408845 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:22.426851 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:22.435684 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:22.463404 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:22.519572 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:22.545439 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:22.545593 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:22.549484 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:22.557603 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:22.571019 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:22.580705 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:22.594182 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:22.600889 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:22.627143 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:22.648322 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:22.654875 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 64274.9us
I20240502 08:25:22.657966 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:22.676457 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:22.680812 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:22.690681 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:22.709825 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:22.790143 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:22.792500 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:22.795012 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:22.841707 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:22.843894 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:22.952602 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:22.963116 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:22.990566 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:23.051661 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:23.096417 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:23.140574 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:23.140574 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:23.144716 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:23.148032 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:23.187170 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:23.214325 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:23.251962 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:23.313709 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:23.326570 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:23.341007 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:23.408337 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:23.415875 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:23.418696 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:25:23.426802 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:23.441855 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:23.466805 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:23.505784 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:25:23.530951 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:23.552379 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:23.655081 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:23.661835 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:23.803200 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:23.874881 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:23.933564 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:23.936825 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 61994.7us
I20240502 08:25:23.975450 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:23.987139 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:24.018903 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:24.031446 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:24.053521 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:24.067272 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:24.142727 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:24.144342 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 08:25:24.145444 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:24.161940 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:24.168581 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:24.190091 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:24.191063 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:24.201126 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:24.237890 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:24.250384 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:24.260277 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:24.277173 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:24.277401 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:24.421448 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 67171.3us
W20240502 08:25:24.425004 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:24.440430 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:24.451267 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:24.470425 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:24.503423 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:24.519852 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:24.522365 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:24.524200 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:24.540511 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:24.557263 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:24.571630 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:24.580093 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:24.583420 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:24.616169 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:25:24.641144 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:24.643918 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:24.651127 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:24.677546 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:24.704437 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:24.710848 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:25.046501 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.053148 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:25.093981 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:25.138746 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:25.170769 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:25.205333 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:25.258245 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:25.268630 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:25.296942 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:25.298091 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:25.312463 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:25.384179 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.391644 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.437044 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 08:25:25.480484 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:25.481309 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:25.496341 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:25:25.498683 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:25.509361 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:25.513880 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:25.518539 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:25.530942 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:25.556568 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:25.580842 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:25.598454 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:25.598968 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 109720us
I20240502 08:25:25.603672 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:25.629484 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:25:25.652338 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:25.669765 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:25.671509 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:25.675858 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:25.705353 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:25.715006 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:25.729589 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 08:25:25.860042 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:25.880683 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.899006 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.912390 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:25.976564 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:25.979988 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:26.090555 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:26.115703 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:26.123590 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:26.150476 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.162176 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:26.176887 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:26.179075 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:26.186168 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:26.222085 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:26.247824 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:26.282807 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:26.329270 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:26.348596 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:25:26.349637 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:26.364272 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:26.441167 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.446931 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:26.517334 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 54188.7us
W20240502 08:25:26.619583 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.628172 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.638687 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.641520 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.678359 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:26.697716 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:26.704991 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:26.712806 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:26.763769 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:26.836468 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:26.859017 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:26.875159 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:26.879590 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:26.896615 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:26.918239 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:26.951213 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:26.955803 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:26.987959 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:27.033360 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:27.045555 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:27.074020 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:27.075309 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:27.166699 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:27.181161 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:27.210099 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:27.222796 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:27.230000 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:27.240474 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:27.274391 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:27.284080 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:25:27.311901 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:27.342080 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:27.412969 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:27.502099 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:27.647367 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:27.648620 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:27.671031 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:27.728689 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:27.739307 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:27.740237 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:27.753015 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:27.759713 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:27.763803 23450 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:27.764191 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:27.801796 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:27.820017 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:27.845930 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:27.851660 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:27.917744 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:27.931358 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:27.995069 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:27.995378 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:28.042090 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:28.047196 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:28.054896 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:28.061897 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:28.138777 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:28.211858 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:28.218386 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:28.227157 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:28.229780 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:28.233294 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:28.248589 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:28.309342 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:28.325383 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:28.444453 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:28.462954 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 08:25:28.464633 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:28.490043 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:28.500052 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:28.525081 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:28.578960 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:28.586047 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:25:28.613775 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:28.675222 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:28.738301 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:28.741526 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:28.756108 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:28.762305 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:28.773273 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:28.806349 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:28.814046 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:28.864357 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:28.876286 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:28.924793 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:28.989629 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.122231 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.123222 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:29.198794 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.205328 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.219878 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.237948 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:29.257687 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:29.329556 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:29.350543 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:29.406646 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:29.447000 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:29.459025 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:29.495589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:29.506799 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 08:25:29.516148 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:29.519773 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:29.547730 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:29.560011 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:29.565524 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:29.568953 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:29.579756 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:29.645568 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:29.693563 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:29.698356 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:29.714061 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:29.719081 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:29.731096 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:29.742193 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:29.742952 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:29.797667 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:29.850929 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:29.968446 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:30.050637 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:30.062739 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:30.071354 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:30.083842 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:30.103648 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:30.119196 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:30.140089 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:30.163043 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:30.181511 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:30.182406 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:30.199012 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:30.200695 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:25:30.206260 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:30.224874 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:30.227679 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:30.232141 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:30.235276 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:30.267316 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:30.299119 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:30.309824 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:30.439541 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:30.445578 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 08:25:30.449632 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:30.527740 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:30.671633 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:30.695930 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:30.751710 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 08:25:30.800809 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:30.823289 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:30.897032 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:25:30.935453 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:31.002015 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:31.009177 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:31.012763 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:31.013803 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.023003 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:31.087505 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.109822 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:31.125247 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:31.146109 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:31.151712 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:31.219179 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:25:31.220642 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:31.234606 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:31.235594 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:31.262542 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:31.267875 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:31.300237 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:31.303926 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:31.313561 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.324343 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.338975 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.350397 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:31.352017 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:31.357522 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:31.425416 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:31.499950 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:31.589044 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:31.595100 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:31.609925 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.611224 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:31.623648 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:25:31.624624 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:31.650416 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:31.661907 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.662590 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:31.678301 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:31.680526 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:31.705637 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:31.737877 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:31.745265 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:31.761476 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:31.773653 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:31.813943 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:31.856389 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:31.875522 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:31.879597 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:31.915313 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:32.114777 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:32.182932 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:32.220113 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:32.380810 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:32.400223 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.429980 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:32.439788 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:32.469828 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:32.486482 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:32.492605 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:32.503258 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:32.503672 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:32.513161 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:32.533802 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:32.542709 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:32.613608 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.762388 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.780894 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:32.793390 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:32.817977 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:32.846033 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:32.851387 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.853847 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:32.862432 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:32.869807 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:32.872045 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:32.878535 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.900832 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:32.901944 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:32.944640 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:32.948122 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:32.947438 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:25:32.999382 23406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 219768us
W20240502 08:25:33.004521 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:33.030777 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:25:33.126401 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:33.132611 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:33.185761 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:33.196822 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:33.242625 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:33.244896 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:33.264663 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:33.286855 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:33.363373 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:33.393211 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:33.458628 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:33.482642 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:25:33.487820 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:33.529393 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:33.611032 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:33.640708 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:33.654855 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:33.700938 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:33.730317 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:33.761222 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:33.763649 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:33.769651 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:33.784935 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:33.806109 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:33.869331 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 44996.1us
W20240502 08:25:33.962870 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:34.072417 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.084434 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:34.085507 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:34.088626 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:34.143787 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:34.150298 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.158581 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:34.162909 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:34.184741 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:34.217955 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:34.245880 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:34.313158 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:34.423427 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:34.430323 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:34.440208 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:34.498435 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:34.643457 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:25:34.657428 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.683573 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:34.687326 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:34.725179 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:34.746168 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:34.750533 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:34.786758 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:34.791985 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:34.820406 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:34.836715 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:34.856501 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.885303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.896122 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:34.900381 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 46860.4us
W20240502 08:25:34.935046 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:34.951099 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:34.969741 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:34.998208 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:34.998890 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:35.001778 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:35.066637 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:35.079710 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:35.100666 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:35.105996 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:35.110190 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:35.112584 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:35.161640 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:35.167527 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:35.227006 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:35.234969 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:35.319929 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:35.334596 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:25:35.337116 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:35.408540 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:35.416352 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:35.418905 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:35.435873 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:35.448019 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:35.453413 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:35.479975 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:25:35.510437 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:25:35.583851 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:35.615199 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:35.824143 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:35.826238 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 08:25:35.827545 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:35.828294 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:35.902215 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:35.905006 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:35.969334 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:35.971017 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:35.971606 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:36.011642 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.039209 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:36.041155 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.091770 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:25:36.202783 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.225456 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:36.226163 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:36.249366 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:36.328563 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:36.330363 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:36.357214 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.364593 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:36.365424 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:36.406631 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:36.409392 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:36.427145 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:36.457047 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:36.464988 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:36.479383 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:36.509516 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:36.510936 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:36.580049 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:36.580307 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:36.734725 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:36.738649 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:36.739429 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:36.744779 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.755136 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:36.755769 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:36.758494 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:36.774574 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:36.782207 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:36.798710 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:36.805120 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:25:36.808832 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:36.826390 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:36.833801 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:36.841933 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:36.846606 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:36.864184 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:36.904795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:36.930397 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:36.935598 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:36.989672 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:37.036123 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:37.188990 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:37.219331 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:37.237197 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:37.321677 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:37.409667 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:37.416039 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:37.441041 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:37.450525 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:37.463030 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:37.480633 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:37.539455 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:37.574486 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:37.577147 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:37.579067 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:37.632220 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:37.644987 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:37.712059 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:37.723886 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:37.729725 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:37.733741 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:37.734519 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:37.742476 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:37.791777 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:37.818264 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:37.820603 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:37.828109 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:37.848553 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:37.866024 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:37.872099 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:37.895642 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:37.932404 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:38.002321 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:38.203441 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:38.212483 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:25:38.298631 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:38.305837 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:38.331946 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:38.340189 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:38.357466 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:38.358021 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:38.371001 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:38.374255 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:38.405045 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:38.408577 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:38.514039 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:25:38.515130 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:25:38.526805 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:38.528332 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:38.544875 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:38.581048 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:38.586470 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:38.627934 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:38.688382 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:38.777835 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:38.779968 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:38.849669 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:38.849758 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:38.856598 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:38.864625 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:38.866088 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:38.923846 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:38.949385 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:38.981304 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:39.032650 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:39.118711 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:39.120077 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:39.166929 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:39.169137 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:39.169616 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:39.203163 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:39.225946 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:39.240058 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:39.280155 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:39.328956 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:39.334030 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:39.340837 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:39.347537 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:39.361258 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:39.404263 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:39.416222 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:39.427325 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:39.433972 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:39.489722 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:39.509718 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:39.552574 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:39.623939 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:39.659641 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:25:39.667690 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:39.757798 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:39.784971 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:40.000156 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:40.005075 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:40.043169 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:40.044468 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:40.052875 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:40.076628 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:40.101595 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:40.123538 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:40.143481 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:40.153965 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:40.173648 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:40.179752 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:40.198685 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:40.230229 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:40.238260 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:40.240006 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:40.246171 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:40.257761 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:40.306229 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:40.309190 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:40.411818 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:40.483748 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:40.519130 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 08:25:40.544368 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:40.577741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:40.645025 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:40.660573 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:25:40.666898 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:40.675091 21487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 140444us
W20240502 08:25:40.692098 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:40.703929 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:40.724826 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:40.799506 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:40.807348 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:40.809218 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:40.834223 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 08:25:40.844122 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:40.848906 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:40.870391 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:41.053154 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:41.104869 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:41.133491 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:41.156478 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:41.177134 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:41.184795 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:41.202581 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:41.224465 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:41.244484 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:41.252050 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:25:41.296263 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:41.298687 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:41.310276 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:41.311015 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:41.354249 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:41.370012 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:25:41.454515 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:41.459532 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:41.467514 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:41.489241 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:41.656821 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:41.764683 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:41.785102 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:41.790679 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:41.851383 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:41.874456 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:25:41.881820 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:41.882763 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:41.882958 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:41.885423 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:41.891007 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:41.930225 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:41.938834 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:41.949519 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:41.956507 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:41.968613 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:41.971699 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:41.990680 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:42.046761 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:42.071807 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:42.089686 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:42.126739 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:42.251359 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:42.287137 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:42.291550 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:42.313318 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:42.332229 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:42.344200 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:42.351193 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:42.359645 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:42.480521 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:42.485443 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:42.499011 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:42.502817 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:42.678697 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:42.690593 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:42.708916 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:42.748838 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:42.769021 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:42.770054 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:42.836750 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:42.841377 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:42.863236 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:42.872771 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:42.885577 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:42.894096 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:42.895839 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:42.903981 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:42.929504 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:42.945420 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:42.949295 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:42.953181 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:42.955345 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:42.962730 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:42.964243 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:42.982355 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:43.110708 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:43.266719 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:43.267153 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:43.277612 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:43.319236 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:43.330278 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:43.345156 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:43.368801 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:25:43.384573 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:43.395195 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:43.443666 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:43.484071 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:43.499366 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:43.613581 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:43.729976 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:43.772658 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:25:43.773084 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:43.791836 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:43.812919 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:25:43.833487 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:43.853396 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:43.855700 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:43.872002 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:43.885488 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:43.928658 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:43.952378 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:44.056617 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:44.122311 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.177513 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:44.179517 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.190685 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:44.220644 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:44.222574 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:44.243602 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:44.252393 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:44.278024 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:44.278246 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:44.279181 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:44.308092 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:44.346585 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.357528 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.366447 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:44.391002 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:44.395936 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:44.473414 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:44.474150 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.475863 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:44.480214 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.484618 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:44.509454 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:44.562672 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:44.663069 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:44.670650 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:44.764348 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:44.780484 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:44.834226 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:44.839354 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:44.843917 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:44.870204 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:44.941931 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 08:25:44.953702 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:44.986352 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:45.006947 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:45.042717 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 87181.9us
I20240502 08:25:45.087698 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:45.093106 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.118592 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:45.192534 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:45.241613 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:45.329701 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:45.368891 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.500205 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:45.501701 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:45.508649 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:45.517490 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.525246 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 08:25:45.548787 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:45.557153 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:45.596803 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.622206 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:25:45.626369 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:45.629379 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:45.678459 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:45.683271 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.696655 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:45.703649 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:45.766794 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:25:45.798533 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:45.844257 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 08:25:45.858474 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:45.874555 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:45.878283 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:45.879714 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:45.881583 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:45.919463 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:45.941504 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:45.945466 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:45.946236 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:45.947929 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:45.960902 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:45.982790 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:45.988853 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:46.017436 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:46.132156 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:46.160651 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:46.190456 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:46.213676 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:46.249718 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:46.286787 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:46.303064 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:46.303508 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:46.305938 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:46.331353 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:46.344228 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:46.488788 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:46.647070 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:46.653044 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:46.657771 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:46.683660 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:46.687932 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:46.728503 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:46.795533 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:25:46.801795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:46.832547 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:46.849004 23806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:33397, user_credentials={real_user=slave}} blocked reactor thread for 53487.4us
W20240502 08:25:46.851714 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:46.907734 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:46.931774 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:25:46.939144 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:46.998456 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:47.002388 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:47.164350 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:47.197917 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:47.216322 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.242352 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:47.256150 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:47.263584 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:47.286854 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:47.294507 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.322495 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:47.397085 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:47.418872 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:47.454805 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:47.523402 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.524559 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:47.558468 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:47.581672 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:47.607522 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.641824 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:47.650029 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.675865 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:47.705991 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:47.710536 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.724459 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:47.763705 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:47.765498 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:47.779780 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:47.786301 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:47.802315 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:47.810127 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 50978.2us
W20240502 08:25:47.848464 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:47.853130 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:47.855988 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:47.882196 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:47.891302 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:47.923781 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:47.929857 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:47.948706 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:47.955927 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:47.963107 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:47.970635 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:48.128108 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:25:48.142612 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:48.208113 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:48.230820 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:48.237124 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:48.258138 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:48.259964 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:48.275692 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:48.279533 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:48.295562 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:48.315729 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:48.412227 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:48.436542 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:48.450574 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:48.463323 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:48.503633 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:48.574802 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:48.620100 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:48.633344 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:48.676831 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:48.709826 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:48.771237 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:48.783226 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:48.823801 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:48.831899 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:25:48.833168 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:25:48.865360 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:48.988485 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:49.050503 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:49.056389 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:49.090821 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:49.095515 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:49.130074 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:49.143986 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:49.266693 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:49.275477 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:49.292824 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:49.336887 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:49.349718 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:49.352308 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:49.379662 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:49.381125 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:49.382728 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:49.402597 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:49.411062 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:49.413794 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:49.488777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:49.522753 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:49.545881 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:49.548774 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:49.553225 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:49.558337 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:49.568524 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:49.600052 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:49.606823 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:49.756649 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:49.792757 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:49.825568 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:49.840026 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:49.841692 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:49.870419 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:49.892127 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:49.921506 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:49.977829 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:50.078562 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:25:50.125735 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:50.142895 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:50.163672 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:50.176990 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:50.205740 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:50.226615 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:50.257409 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:50.270884 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:50.278443 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:50.302474 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:50.305097 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:50.386683 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:50.477679 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:50.481281 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:50.492043 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:50.519922 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:50.568082 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:50.577036 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:50.583427 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:25:50.584911 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:50.603611 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:50.632757 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:50.634333 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:50.648068 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:50.648412 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:50.653226 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:50.690637 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:50.729019 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:50.738963 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:25:50.761092 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:50.764649 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:50.773571 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:50.829937 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:50.837191 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:50.867985 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:50.872835 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:50.877086 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:50.905467 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 08:25:50.920102 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:50.937104 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:50.952989 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:50.954963 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:50.963506 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:50.990909 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:51.033071 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:51.082746 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:25:51.093917 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:51.264250 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:51.329542 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:51.376291 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:51.380590 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:51.383414 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:51.401820 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:51.453717 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:51.499672 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:51.502959 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:51.510118 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:51.512961 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:51.626075 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:51.662309 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:51.723127 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:51.723848 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:51.767365 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:51.770027 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:51.785511 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:51.787801 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:51.846973 21879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 44038.2us
I20240502 08:25:51.861240 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:51.890710 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:51.894836 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:51.935274 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 08:25:51.936591 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:52.022125 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:52.055086 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:52.102066 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:52.107134 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:52.111930 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:52.114753 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:25:52.118499 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:52.134469 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:52.149799 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:52.258091 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:52.333611 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:52.430256 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:52.455511 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:52.463814 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:52.492450 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:52.509929 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:52.524319 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:52.588187 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:52.620429 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:52.652137 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:52.786231 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:52.796555 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:52.798527 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:52.859666 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:52.911307 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:52.933694 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:52.940584 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:52.945878 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:52.968170 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:52.981493 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:52.994014 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:52.998944 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:53.034726 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:53.077899 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:53.121429 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:53.185915 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:53.206800 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:53.245705 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:53.254722 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:53.255795 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:53.286708 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:25:53.300053 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:53.301337 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:53.305498 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:53.318869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:53.390237 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:53.419454 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:53.422502 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:53.454533 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:53.492148 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:53.508567 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:53.512544 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:53.531522 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:53.542675 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:53.581517 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:53.588100 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:53.624728 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:53.742262 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:53.749521 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:25:53.771550 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:53.777171 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:53.789016 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:53.796866 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:53.797186 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:53.838554 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:53.900215 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:25:53.983762 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:54.024080 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:54.126005 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:54.163961 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:54.170647 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:54.196353 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:54.312424 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:54.326438 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:54.326982 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.357815 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.392616 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:54.399061 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:54.415234 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:54.447845 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:54.496165 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:54.514801 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:54.530791 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:54.540221 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:54.556612 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.557780 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:54.570147 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:54.596433 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:54.643121 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.686476 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:54.714700 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.719529 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:54.736996 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:54.773118 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:54.786851 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:54.826524 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:54.885027 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:54.966634 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:54.999281 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:55.010659 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:55.037695 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:55.078476 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:55.080639 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:55.089912 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 08:25:55.091153 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:55.148626 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:55.195452 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.237098 21748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 57340.2us
W20240502 08:25:55.255165 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:55.310438 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:55.372278 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:55.372402 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:55.412442 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:55.415812 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:55.435698 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:55.460243 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:55.480082 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:55.482146 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.516109 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.518153 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:55.547128 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:55.589643 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:55.621206 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 08:25:55.732758 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:25:55.797878 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.799405 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.835161 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:55.870030 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:55.898030 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:55.960659 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 08:25:55.973129 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:56.018906 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:56.022615 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:56.040635 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:56.054989 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:56.095621 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:25:56.096244 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:56.138463 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:56.163197 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:56.193360 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:56.204869 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:56.222258 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:56.260092 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:56.267019 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:56.281271 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:56.298094 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:56.318563 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:56.333771 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:56.341629 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:56.362342 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:56.366640 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:56.366819 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:56.368681 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:56.379648 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:56.415398 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:56.428393 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:56.439307 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:56.478217 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:56.542234 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:56.552345 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:56.560530 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:56.613134 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:56.618887 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:56.621301 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:56.651109 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:56.688202 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:56.765084 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:56.868866 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:57.007383 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:57.060248 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:57.065874 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:57.067428 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:57.078733 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:57.102105 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:57.106629 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 08:25:57.126655 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:57.131778 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:57.244977 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:57.310400 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:57.326599 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:57.397994 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:25:57.473044 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:57.513670 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:57.581626 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:57.635076 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:57.650278 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:57.696547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:57.698869 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:57.727635 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:57.732883 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:57.741715 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:57.786788 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:57.793149 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:57.848604 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:57.858654 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:57.881907 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:57.889673 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:57.905223 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:57.914948 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:57.925663 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:57.938403 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:57.950215 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.024353 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.088330 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:58.089735 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:58.262042 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:58.269064 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:25:58.277415 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:58.284153 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:58.313773 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:58.343848 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:58.348228 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:25:58.373781 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:58.401235 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:58.406675 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:58.412763 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:25:58.494969 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:58.515782 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:58.520231 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:58.534170 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:58.539837 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:58.592425 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:58.670393 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.699553 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.707978 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:58.732843 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:58.745584 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:58.762051 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:58.796427 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:25:58.807117 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.853680 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:58.909107 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:58.979027 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:58.979363 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:58.999912 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:59.015413 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:59.036338 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:25:59.039844 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:59.076525 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:59.142274 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:59.168848 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:25:59.183808 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:25:59.240377 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:25:59.251271 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:59.252233 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:25:59.265364 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:25:59.296090 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:25:59.296842 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:59.327807 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:59.415649 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:59.543921 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:59.591130 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:59.623091 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:25:59.628612 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:59.633273 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:25:59.654100 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:25:59.664403 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:25:59.678912 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:25:59.690109 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:25:59.719533 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:25:59.766808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:59.834924 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:25:59.854667 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:25:59.871541 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:25:59.890324 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:25:59.924989 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:25:59.954860 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:00.058126 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:00.151079 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 08:26:00.183887 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:00.184978 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:00.193475 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:00.224288 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:26:00.238551 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:00.238721 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:00.243063 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:00.247056 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:00.255792 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:00.256650 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:00.275343 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:00.363646 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:00.441705 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:00.502614 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:00.535948 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:00.644306 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:00.651849 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 08:26:00.667671 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:00.717458 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:00.724686 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:00.727406 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:00.744848 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:26:00.750622 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:00.771562 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:00.772171 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:00.897207 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 82910.2us
I20240502 08:26:00.901381 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:00.923170 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:00.926421 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:00.936789 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:26:00.948436 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:00.962325 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:00.977739 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 08:26:00.983534 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:26:00.987187 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:00.989274 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:01.036386 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:01.046555 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:01.060726 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:01.064311 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:01.068248 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:01.086410 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:26:01.097465 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:01.104266 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:01.118294 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:01.144001 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:01.150178 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:01.155826 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:01.170658 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:01.237659 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:01.258206 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:01.278895 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:01.312623 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:01.344170 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:01.350919 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:01.397490 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:01.397936 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:01.418041 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:01.452718 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:01.503083 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:01.593080 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:01.624481 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:01.759306 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:01.786886 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:01.887549 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:01.896024 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:01.959307 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:01.963214 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:02.111752 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:02.118422 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:02.131682 23405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 08:26:02.132900 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:02.182461 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:02.194245 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:02.194527 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:26:02.224440 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:02.276136 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:02.292865 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:02.299600 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:02.346877 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:02.353303 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:02.368424 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:02.523363 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:02.526347 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:02.530467 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:02.552227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:02.564455 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:02.578377 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:02.589596 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:02.597432 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:02.614805 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:02.651106 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:02.666448 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:02.689110 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:02.810531 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:02.848148 23806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:26:02.856910 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:02.862721 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:02.897532 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:02.903671 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:02.921578 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:02.942543 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:02.947299 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:02.985605 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:03.006844 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:03.072213 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:03.109028 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:03.187947 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:03.195475 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:03.200471 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:26:03.235136 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:03.265679 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:03.279752 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:03.304422 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:03.321790 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:26:03.376863 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:03.481274 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:03.538233 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:03.538614 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:03.577095 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:03.635865 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:03.644649 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:03.647619 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:03.654642 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:03.655817 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:03.682636 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:03.686949 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:03.691184 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:03.763273 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:03.792657 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:03.832566 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:03.859606 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:03.878782 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:03.888273 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:03.896447 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:03.901719 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:03.905529 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:03.931677 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:26:04.000715 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:26:04.066287 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:04.080686 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:04.112205 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:26:04.236049 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:04.266379 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:04.275630 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:04.305645 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:04.323343 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:04.331569 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:04.335179 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:04.338450 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:04.341300 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:04.434744 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:04.676453 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:04.687119 21748 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:04.689786 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:04.714617 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:04.735759 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:04.826858 21618 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:04.833683 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:04.877139 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:04.888113 21748 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:04.969075 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:04.981773 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:04.989384 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:05.016789 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:05.024770 21618 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:05.026496 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:05.038574 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:05.094563 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:05.175114 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:05.175727 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:05.186472 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:05.231907 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:05.249925 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:26:05.270265 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:05.274194 21617 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:05.294898 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:05.346766 21747 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:05.396939 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:05.400928 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:05.425730 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:05.533780 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:05.568444 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:05.596385 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:05.647349 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:05.670007 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 08:26:05.676328 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:05.698105 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:05.708614 21748 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:05.712860 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:05.724017 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:05.743786 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:05.806787 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:05.812952 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:05.850520 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:05.890259 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:05.900012 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:05.957485 21747 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:26:05.967481 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:05.969007 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:05.972309 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:06.061818 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 08:26:06.084468 21618 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:06.120011 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:26:06.184124 21617 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:06.198365 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:06.222471 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:06.223165 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:06.247325 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:06.248507 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:06.299504 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:06.300065 21747 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:06.329450 21879 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:06.473634 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:26:06.501175 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:26:06.509155 21486 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:26:06.518652 9911 raft_consensus.cc:998] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.301s)
W20240502 08:26:06.522023 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:06.522763 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:06.522867 9911 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:26:06.570250 21552 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:06.571187 10113 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:26:06.600445 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:06.607831 10113 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:06.612201 21549 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:06.619035 21408 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
W20240502 08:26:06.698485 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:06.701958 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:06.779516 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:06.780074 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:26:06.785353 10124 raft_consensus.cc:998] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.006s)
I20240502 08:26:06.786183 10124 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:06.821452 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:06.839226 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:06.905035 9906 raft_consensus.cc:998] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.159s)
I20240502 08:26:06.906571 10182 raft_consensus.cc:998] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.166s)
I20240502 08:26:06.907519 10182 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:06.909777 9906 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:06.910794 23870 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:06.911782 9978 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:06.916329 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:06.934382 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 50535.8us
I20240502 08:26:06.956027 23868 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:06.955019 9978 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:06.963307 23471 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:06.965785 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:06.967352 21409 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:06.993479 21812 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.003119 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:07.014144 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:07.015231 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:07.015619 21942 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:07.019357 23870 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:07.019907 10181 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.020054 10124 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.023152 10124 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.028018 23870 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.030175 10183 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.031944 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cec56facd7f740aeb42bc7c32495dae3 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 08:26:07.034010 21617 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:07.035710 21409 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.040480 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:07.040920 10180 raft_consensus.cc:998] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.368s)
I20240502 08:26:07.041783 10180 consensus_queue.cc:237] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
W20240502 08:26:07.056454 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.067312 9978 raft_consensus.cc:998] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.266s)
I20240502 08:26:07.068048 9978 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:07.082019 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:07.082634 23405 consensus_peers.cc:462] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:07.093957 23868 raft_consensus.cc:1261] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:26:07.094293 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.098448 10182 consensus_queue.cc:1035] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:26:07.107112 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:26:07.118431 10182 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.119331 23868 raft_consensus.cc:2932] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
W20240502 08:26:07.126883 21880 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Couldn't send request to peer cfff665685a14d8796aece26addab59c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec56facd7f740aeb42bc7c32495dae3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.127286 21409 catalog_manager.cc:5484] T 182a6f2df91242cfa10f05ef788a19cb P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.132644 23471 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:07.138917 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:07.157253 23470 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:07.157940 10181 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.160607 10124 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:07.163211 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:07.184957 21409 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.186838 10181 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:26:07.186638 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 44624.4us
I20240502 08:26:07.192337 23470 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.200310 10113 raft_consensus.cc:998] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.199s)
I20240502 08:26:07.201800 10113 consensus_queue.cc:237] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:07.209609 10198 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:26:07.216112 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:26:07.216058 23470 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.225117 21683 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:26:07.226796 10113 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:26:07.233675 21812 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.234359 21409 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.239624 10221 raft_consensus.cc:998] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.427s)
I20240502 08:26:07.245055 10221 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:07.246968 5897 raft_consensus.cc:998] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.156s)
I20240502 08:26:07.249150 10181 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.249485 5897 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:07.253685 10180 raft_consensus.cc:998] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.348s)
I20240502 08:26:07.254550 10180 consensus_queue.cc:237] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:26:07.257637 10222 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.260146 21942 consensus_queue.cc:237] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:07.267117 10113 raft_consensus.cc:998] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.448s)
W20240502 08:26:07.267529 21880 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Couldn't send request to peer cfff665685a14d8796aece26addab59c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f5c2dfcd974ee2b8720e9eb052c3f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.268060 10113 consensus_queue.cc:237] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } }
I20240502 08:26:07.268754 23470 raft_consensus.cc:1261] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:07.269465 9978 consensus_queue.cc:1035] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.273710 9978 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.274619 21410 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.278849 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:07.279466 23470 raft_consensus.cc:2932] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.284473 23469 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.286124 21683 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.294723 21682 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:07.294683 23471 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:07.301853 21409 catalog_manager.cc:5484] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.304561 10227 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:07.305442 10227 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:07.325417 21812 consensus_queue.cc:237] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
W20240502 08:26:07.327821 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.328428 23868 raft_consensus.cc:1261] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:26:07.328985 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:26:07.330037 10241 raft_consensus.cc:998] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.352s)
I20240502 08:26:07.330873 10226 consensus_queue.cc:1035] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.331152 10241 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:26:07.331847 21749 consensus_peers.cc:485] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b3f6a0b2561419ca6bd832fd65e3196. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.337149 21681 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.338588 10227 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.344352 21682 raft_consensus.cc:1261] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.346179 10113 consensus_queue.cc:1035] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.352218 10113 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:07.352500 21682 raft_consensus.cc:2932] T 7b3f6a0b2561419ca6bd832fd65e3196 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:07.353394 21549 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:26:07.354113 10124 raft_consensus.cc:998] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Attempting to remove follower fd5e97ae7e344e4687b15ca0cde8d0ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer fd5e97ae7e344e4687b15ca0cde8d0ba for more than 300 seconds (300.406s)
I20240502 08:26:07.354142 10241 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.354993 10124 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:07.359733 21410 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 8 to 10, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 2 leader_uuid: "95f32202b9db457fb09872f6f51ea8ba" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.361294 21409 catalog_manager.cc:5484] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:07.375304 10240 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.380568 21552 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.386317 21942 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.387293 10222 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.393704 23868 raft_consensus.cc:1261] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:07.395577 9906 consensus_queue.cc:1035] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:26:07.397532 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.410692 10222 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.412735 21942 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.420131 23471 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
W20240502 08:26:07.451581 21746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 122473us
I20240502 08:26:07.452836 23870 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.455578 21405 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.457583 21408 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.476097 21409 catalog_manager.cc:5484] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.478947 10226 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.487047 10180 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.487672 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:07.487674 23471 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:07.492596 23868 raft_consensus.cc:2932] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.495467 9978 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:07.517774 9978 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.519699 10226 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.525194 23471 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.527400 21410 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 7 to 8, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.529987 21409 catalog_manager.cc:5484] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.546597 23871 raft_consensus.cc:2932] T dc49c050280346db88d1323e30513c80 P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.561482 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:07.563544 10267 ts_tablet_manager.cc:902] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:26:07.566152 23870 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.573520 10240 raft_consensus.cc:998] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.164s)
I20240502 08:26:07.574623 10240 consensus_queue.cc:237] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:07.588799 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:07.589461 21942 consensus_queue.cc:237] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:07.592133 23471 raft_consensus.cc:1261] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:26:07.592221 21880 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Couldn't send request to peer cfff665685a14d8796aece26addab59c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.592836 9978 consensus_queue.cc:1035] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.598886 9978 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.600487 9906 raft_consensus.cc:998] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.170s)
I20240502 08:26:07.601289 9906 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:07.602656 23471 raft_consensus.cc:2932] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.608656 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:07.613215 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7b3f6a0b2561419ca6bd832fd65e3196 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:07.616449 21410 catalog_manager.cc:5484] T a20b2fd9f1ce488ba7793ed7ac268c16 P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 8 to 9, VOTER fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } }
I20240502 08:26:07.621053 23868 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:07.621778 9286 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:07.622391 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:07.629962 21549 raft_consensus.cc:1261] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:07.630872 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:07.632541 10240 consensus_queue.cc:1035] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:07.642684 21408 catalog_manager.cc:5484] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 8 to 9, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:26:07.650151 23471 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:07.654086 21551 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Refusing update from remote peer 20b637fbec5444dd91b3af9cdda482f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:07.656334 9286 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:07.658454 21550 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.658672 23470 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.659538 10180 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.660218 10180 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:26:07.664232 23406 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Couldn't send request to peer 13e764b27ed449c09aee719cdb2597f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.663962 9286 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.670536 21550 raft_consensus.cc:2932] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.675673 10267 tablet_copy_client.cc:308] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:26:07.676872 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet cec56facd7f740aeb42bc7c32495dae3 from peer cfff665685a14d8796aece26addab59c ({username='slave'} at 127.8.68.193:53335)
I20240502 08:26:07.677054 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet cec56facd7f740aeb42bc7c32495dae3 from peer cfff665685a14d8796aece26addab59c at {username='slave'} at 127.8.68.193:53335: session id = cfff665685a14d8796aece26addab59c-cec56facd7f740aeb42bc7c32495dae3
I20240502 08:26:07.680284 21408 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 9 to 10, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.680393 10241 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.681113 21964 tablet_copy_source_session.cc:205] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:07.686002 21550 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.688849 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 12c17796e5d04014aafdac46984f4ba9 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:07.691895 10180 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.693295 23469 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.710865 21551 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.712440 21410 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:26:07.716920 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a20b2fd9f1ce488ba7793ed7ac268c16: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:07.719412 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:07.721891 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:07.740279 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:07.741919 23868 raft_consensus.cc:2932] T 203e7914da3f4494b239c6603513caee P 95f32202b9db457fb09872f6f51ea8ba [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:07.753926 21408 catalog_manager.cc:5484] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 2 leader_uuid: "20b637fbec5444dd91b3af9cdda482f5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
W20240502 08:26:07.768249 21486 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:07.770999 10289 ts_tablet_manager.cc:902] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:26:07.772696 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:07.782956 9906 raft_consensus.cc:998] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.189s)
I20240502 08:26:07.783754 9906 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:07.793524 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:07.795317 21812 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.796133 10183 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.802234 21942 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.803126 10183 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.813850 10242 raft_consensus.cc:998] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.233s)
I20240502 08:26:07.814647 10242 consensus_queue.cc:237] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:07.825819 10183 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.828625 21943 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.842006 23471 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.842731 10221 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.846293 21410 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.851913 21943 raft_consensus.cc:1261] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:07.852623 10221 consensus_queue.cc:1035] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:07.854085 10288 raft_consensus.cc:998] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.028s)
I20240502 08:26:07.854871 10288 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:07.857381 21812 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:07.857863 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:07.865869 23469 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:07.866544 10288 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:07.865233 10242 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.869372 23470 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.871752 21943 raft_consensus.cc:2932] T d8042f6ab6404c5a9f8cce0b8f971542 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.876130 21409 catalog_manager.cc:5484] T d8042f6ab6404c5a9f8cce0b8f971542 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 6 to 7, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.887558 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:07.897804 9871 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.902345 10308 ts_tablet_manager.cc:902] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:26:07.904407 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:07.906646 23470 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.920588 21411 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:07.935775 21550 consensus_queue.cc:237] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:07.953553 10289 tablet_copy_client.cc:308] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:26:07.953783 23470 raft_consensus.cc:1261] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:26:07.954058 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f5d97e6e0a4755be41699246907e55. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:07.954474 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer cfff665685a14d8796aece26addab59c ({username='slave'} at 127.8.68.193:53335)
I20240502 08:26:07.954643 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 from peer cfff665685a14d8796aece26addab59c at {username='slave'} at 127.8.68.193:53335: session id = cfff665685a14d8796aece26addab59c-07f5c2dfcd974ee2b8720e9eb052c3f7
I20240502 08:26:07.954813 9871 consensus_queue.cc:1035] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:07.958307 21964 tablet_copy_source_session.cc:205] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:07.960975 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:07.970454 10289 tablet_copy_client.cc:783] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:07.983420 10183 raft_consensus.cc:998] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.174s)
I20240502 08:26:07.984148 10183 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:07.989418 10289 tablet_copy_client.cc:647] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:07.992069 10289 tablet_copy_client.cc:523] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:07.998131 10289 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:26:08.012063 10310 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.017019 23469 raft_consensus.cc:2932] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.020252 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 59f5d97e6e0a4755be41699246907e55 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:08.030150 21410 catalog_manager.cc:5484] T 59f5d97e6e0a4755be41699246907e55 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.030630 21943 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:26:08.031335 9906 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:26:08.047327 9906 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.054955 21943 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.061705 21410 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.063470 10226 raft_consensus.cc:998] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.268s)
I20240502 08:26:08.064188 10226 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:08.081962 10267 tablet_copy_client.cc:783] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.085175 21682 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
W20240502 08:26:08.088130 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a45b1da348e4bb89f021df103a0603f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:08.090030 10289 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.090200 21942 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:08.090626 10289 tablet_bootstrap.cc:492] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Bootstrap complete.
I20240502 08:26:08.090833 10226 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:08.091480 10289 ts_tablet_manager.cc:1367] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.093s user 0.020s sys 0.011s
I20240502 08:26:08.092536 10289 raft_consensus.cc:348] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.092836 10289 raft_consensus.cc:729] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.093602 10289 consensus_queue.cc:260] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.094013 10308 tablet_copy_client.cc:308] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:26:08.094585 10289 ts_tablet_manager.cc:1398] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:26:08.095860 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session cfff665685a14d8796aece26addab59c-07f5c2dfcd974ee2b8720e9eb052c3f7 received from {username='slave'} at 127.8.68.193:53335
I20240502 08:26:08.095880 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 7b3f6a0b2561419ca6bd832fd65e3196 from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:57225)
I20240502 08:26:08.095999 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session cfff665685a14d8796aece26addab59c-07f5c2dfcd974ee2b8720e9eb052c3f7 on tablet 07f5c2dfcd974ee2b8720e9eb052c3f7 with peer cfff665685a14d8796aece26addab59c
I20240502 08:26:08.096027 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 7b3f6a0b2561419ca6bd832fd65e3196 from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:57225: session id = 0f655ab8f20d4ba482c3173f9594b98b-7b3f6a0b2561419ca6bd832fd65e3196
I20240502 08:26:08.093745 21943 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:08.097262 10183 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:26:08.100970 10183 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.103699 21944 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.104327 21833 tablet_copy_source_session.cc:205] T 7b3f6a0b2561419ca6bd832fd65e3196 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.106912 10289 ts_tablet_manager.cc:902] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:26:08.112283 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f6a0b2561419ca6bd832fd65e3196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:08.115875 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8a45b1da348e4bb89f021df103a0603f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:08.117028 10226 raft_consensus.cc:998] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.384s)
I20240502 08:26:08.117689 10267 tablet_copy_client.cc:647] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.117812 10226 consensus_queue.cc:237] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:08.118453 21410 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 6 to 7, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.122258 10267 tablet_copy_client.cc:523] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.137708 21552 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:26:08.138417 10226 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:26:08.150135 10242 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:08.151033 21552 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:08.156111 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7): 4925 bytes on disk
I20240502 08:26:08.156862 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(07f5c2dfcd974ee2b8720e9eb052c3f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 08:26:08.158648 21410 catalog_manager.cc:5484] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 5 to 6, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:08.162959 10267 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:26:08.163060 10242 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.170048 21942 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.173849 10308 tablet_copy_client.cc:783] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.184229 21409 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.191531 9871 raft_consensus.cc:998] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.312s)
I20240502 08:26:08.192644 9871 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:08.198989 21811 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:08.199226 21813 consensus_queue.cc:237] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:08.201581 21942 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:08.202301 10221 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:26:08.202817 21746 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 852613f14b324b38b5eecc29d4c602cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:08.202860 21552 raft_consensus.cc:1261] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:26:08.204644 10221 consensus_queue.cc:1035] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:26:08.206190 21749 consensus_peers.cc:485] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:08.206920 10308 tablet_copy_client.cc:647] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.207369 10221 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.213272 21942 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.213467 10242 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.215111 10308 tablet_copy_client.cc:523] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.216409 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 852613f14b324b38b5eecc29d4c602cd with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:08.218151 21552 raft_consensus.cc:2932] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.229007 21942 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:26:08.229713 10310 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:26:08.230309 21409 catalog_manager.cc:5484] T 296ccf9be99545789bbb3ff4031d3170 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 6 to 7, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.233258 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 296ccf9be99545789bbb3ff4031d3170 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:08.239482 10327 ts_tablet_manager.cc:902] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Initiating tablet copy from peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:26:08.242192 21410 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.242754 23870 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:26:08.243638 10310 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:26:08.255098 10310 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.260517 23870 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.272930 21942 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.276991 21409 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 95f32202b9db457fb09872f6f51ea8ba reported cstate change: config changed from index 10 to 11, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:08.293862 10308 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:26:08.325744 10289 tablet_copy_client.cc:308] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:26:08.331683 10327 tablet_copy_client.cc:308] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.197:45373
I20240502 08:26:08.332474 23491 tablet_copy_service.cc:140] P 20b637fbec5444dd91b3af9cdda482f5: Received BeginTabletCopySession request for tablet 12c17796e5d04014aafdac46984f4ba9 from peer 13e764b27ed449c09aee719cdb2597f8 ({username='slave'} at 127.8.68.195:45899)
I20240502 08:26:08.332650 23491 tablet_copy_service.cc:161] P 20b637fbec5444dd91b3af9cdda482f5: Beginning new tablet copy session on tablet 12c17796e5d04014aafdac46984f4ba9 from peer 13e764b27ed449c09aee719cdb2597f8 at {username='slave'} at 127.8.68.195:45899: session id = 13e764b27ed449c09aee719cdb2597f8-12c17796e5d04014aafdac46984f4ba9
I20240502 08:26:08.341742 23491 tablet_copy_source_session.cc:205] T 12c17796e5d04014aafdac46984f4ba9 P 20b637fbec5444dd91b3af9cdda482f5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.347018 10327 ts_tablet_manager.cc:1875] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:26:08.348155 21552 raft_consensus.cc:1203] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:26:08.376031 10327 ts_tablet_manager.cc:1888] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240502 08:26:08.376497 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c17796e5d04014aafdac46984f4ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:08.383845 21982 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:57184 (request call id 6) took 1060 ms. Trace:
I20240502 08:26:08.384205 10327 tablet_copy_client.cc:783] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.384001 21982 rpcz_store.cc:276] 0502 08:26:07.323450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:26:07.323516 (+ 66us) service_pool.cc:225] Handling call
0502 08:26:08.383799 (+1060283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:26:08.395746 10267 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.396185 10267 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Bootstrap complete.
I20240502 08:26:08.396490 10267 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.234s user 0.024s sys 0.004s
I20240502 08:26:08.397301 10267 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.397550 10267 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.401455 10327 tablet_copy_client.cc:647] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.403687 10327 tablet_copy_client.cc:523] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.397807 10267 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.406486 10267 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:26:08.409745 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3): 4840 bytes on disk
I20240502 08:26:08.409771 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer cfff665685a14d8796aece26addab59c ({username='slave'} at 127.8.68.193:53335)
I20240502 08:26:08.409950 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 from peer cfff665685a14d8796aece26addab59c at {username='slave'} at 127.8.68.193:53335: session id = cfff665685a14d8796aece26addab59c-76bb1e3e2b9d4e31bf93ac1ce08a7833
I20240502 08:26:08.410526 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 08:26:08.414582 21964 tablet_copy_source_session.cc:205] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.425463 10327 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
I20240502 08:26:08.429482 10289 ts_tablet_manager.cc:1875] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:26:08.430325 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session cfff665685a14d8796aece26addab59c-cec56facd7f740aeb42bc7c32495dae3 received from {username='slave'} at 127.8.68.193:53335
I20240502 08:26:08.430456 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session cfff665685a14d8796aece26addab59c-cec56facd7f740aeb42bc7c32495dae3 on tablet cec56facd7f740aeb42bc7c32495dae3 with peer cfff665685a14d8796aece26addab59c
I20240502 08:26:08.443230 10289 ts_tablet_manager.cc:1888] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:26:08.443636 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:08.462489 10289 tablet_copy_client.cc:783] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.467957 10308 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.468420 10308 tablet_bootstrap.cc:492] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:26:08.468747 10308 ts_tablet_manager.cc:1367] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.175s user 0.023s sys 0.010s
I20240502 08:26:08.469568 10308 raft_consensus.cc:348] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:08.473438 10308 raft_consensus.cc:729] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.473541 10289 tablet_copy_client.cc:647] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.474027 10308 consensus_queue.cc:260] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:08.476624 10289 tablet_copy_client.cc:523] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.483340 10289 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
I20240502 08:26:08.486927 10308 ts_tablet_manager.cc:1398] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:26:08.487744 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-7b3f6a0b2561419ca6bd832fd65e3196 received from {username='slave'} at 127.8.68.196:57225
I20240502 08:26:08.487916 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-7b3f6a0b2561419ca6bd832fd65e3196 on tablet 7b3f6a0b2561419ca6bd832fd65e3196 with peer 0f655ab8f20d4ba482c3173f9594b98b
I20240502 08:26:08.506536 10327 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.509073 10327 tablet_bootstrap.cc:492] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap complete.
I20240502 08:26:08.509557 10327 ts_tablet_manager.cc:1367] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.084s user 0.020s sys 0.008s
I20240502 08:26:08.510545 10327 raft_consensus.cc:348] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:08.510883 10327 raft_consensus.cc:729] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.511308 10327 consensus_queue.cc:260] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:08.512552 10327 ts_tablet_manager.cc:1398] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:26:08.515879 23491 tablet_copy_service.cc:342] P 20b637fbec5444dd91b3af9cdda482f5: Request end of tablet copy session 13e764b27ed449c09aee719cdb2597f8-12c17796e5d04014aafdac46984f4ba9 received from {username='slave'} at 127.8.68.195:45899
I20240502 08:26:08.516031 23491 tablet_copy_service.cc:433] P 20b637fbec5444dd91b3af9cdda482f5: ending tablet copy session 13e764b27ed449c09aee719cdb2597f8-12c17796e5d04014aafdac46984f4ba9 on tablet 12c17796e5d04014aafdac46984f4ba9 with peer 13e764b27ed449c09aee719cdb2597f8
I20240502 08:26:08.545809 21552 raft_consensus.cc:1203] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:26:08.560791 10391 ts_tablet_manager.cc:902] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:26:08.577791 10390 ts_tablet_manager.cc:902] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
W20240502 08:26:08.595738 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:26:08.600448 10391 tablet_copy_client.cc:308] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:26:08.601341 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet 8a45b1da348e4bb89f021df103a0603f from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:42037)
I20240502 08:26:08.601580 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet 8a45b1da348e4bb89f021df103a0603f from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:42037: session id = 95f32202b9db457fb09872f6f51ea8ba-8a45b1da348e4bb89f021df103a0603f
I20240502 08:26:08.607383 21703 tablet_copy_source_session.cc:205] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.611390 10391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a45b1da348e4bb89f021df103a0603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:08.622071 10391 tablet_copy_client.cc:783] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.632601 10391 tablet_copy_client.cc:647] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.636214 10391 tablet_copy_client.cc:523] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.640122 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196): 4925 bytes on disk
I20240502 08:26:08.640867 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(7b3f6a0b2561419ca6bd832fd65e3196) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:26:08.642350 10289 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.642364 10391 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:26:08.642899 10289 tablet_bootstrap.cc:492] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Bootstrap complete.
I20240502 08:26:08.651104 10390 tablet_copy_client.cc:308] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:26:08.657509 10289 ts_tablet_manager.cc:1367] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.175s user 0.044s sys 0.008s
I20240502 08:26:08.659970 10289 raft_consensus.cc:348] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.660321 10289 raft_consensus.cc:729] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.660634 10289 consensus_queue.cc:260] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:26:08.665127 10289 ts_tablet_manager.cc:1398] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:26:08.667881 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9): 4840 bytes on disk
I20240502 08:26:08.668601 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(12c17796e5d04014aafdac46984f4ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:26:08.711035 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session cfff665685a14d8796aece26addab59c-76bb1e3e2b9d4e31bf93ac1ce08a7833 received from {username='slave'} at 127.8.68.193:53335
I20240502 08:26:08.711177 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session cfff665685a14d8796aece26addab59c-76bb1e3e2b9d4e31bf93ac1ce08a7833 on tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833 with peer cfff665685a14d8796aece26addab59c
I20240502 08:26:08.743620 10308 ts_tablet_manager.cc:902] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:08.749390 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:26:08.751353 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:08.757829 21812 raft_consensus.cc:1203] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 08:26:08.761734 23870 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:08.800504 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:08.809275 23469 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:26:08.815503 10235 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:26:08.824848 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:08.834445 10446 ts_tablet_manager.cc:902] T 852613f14b324b38b5eecc29d4c602cd P 95f32202b9db457fb09872f6f51ea8ba: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
F20240502 08:26:08.834827 10446 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.queue_time_us, tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.queue_time_us) (tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.run_cpu_time_us, tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.run_cpu_time_us) (tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.run_wall_time_us, tablet-download-pool-1648586b889b4a1e9ac34481cec6257a.run_wall_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.queue_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_cpu_time_us) (tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us, tablet-download-pool-182a6f2df91242cfa10f05ef788a19cb.run_wall_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.run_cpu_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.run_cpu_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.run_wall_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.run_wall_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.queue_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.queue_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_cpu_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_cpu_time_us) (tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_wall_time_us, tablet-download-pool-2de41d6c6de0400397645c05a0fa17de.run_wall_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us) (tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.queue_time_us, tablet-download-pool-852613f14b324b38b5eecc29d4c602cd.queue_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.queue_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.queue_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_cpu_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_cpu_time_us) (tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_wall_time_us, tablet-download-pool-8a45b1da348e4bb89f021df103a0603f.run_wall_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.queue_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.queue_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_cpu_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_cpu_time_us) (tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_wall_time_us, tablet-download-pool-9c0a886b3ba04737a6988c3b58c9698e.run_wall_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.queue_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_cpu_time_us) (tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us, tablet-download-pool-a20b2fd9f1ce488ba7793ed7ac268c16.run_wall_time_us) (tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.queue_time_us, tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.queue_time_us) (tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.run_cpu_time_us, tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.run_cpu_time_us) (tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.run_wall_time_us, tablet-download-pool-c4862ea2f67c43a28689b464d8ec96e6.run_wall_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.queue_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.queue_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_cpu_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_cpu_time_us) (tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_wall_time_us, tablet-download-pool-cec56facd7f740aeb42bc7c32495dae3.run_wall_time_us) (tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.queue_time_us, tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.queue_time_us) (tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.run_cpu_time_us, tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.run_cpu_time_us) (tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.run_wall_time_us, tablet-download-pool-d62f6eec9ee04f93a3738a83f790292c.run_wall_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.queue_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_cpu_time_us) (tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us, tablet-download-pool-dc49c050280346db88d1323e30513c80.run_wall_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.queue_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.queue_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_cpu_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_cpu_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_wall_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 08:26:08.839217 10308 tablet_copy_client.cc:308] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:26:08.841929 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 296ccf9be99545789bbb3ff4031d3170 from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:57225)
I20240502 08:26:08.842121 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 296ccf9be99545789bbb3ff4031d3170 from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:57225: session id = 0f655ab8f20d4ba482c3173f9594b98b-296ccf9be99545789bbb3ff4031d3170
I20240502 08:26:08.847411 21552 raft_consensus.cc:1203] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:26:08.848187 21833 tablet_copy_source_session.cc:205] T 296ccf9be99545789bbb3ff4031d3170 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.851080 10308 ts_tablet_manager.cc:1875] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7fabb43bedcd google::LogMessage::Fail() at ??:0
I20240502 08:26:08.861045 10451 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.862274 10391 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:08.862272 23469 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.862681 10391 tablet_bootstrap.cc:492] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap complete.
I20240502 08:26:08.863045 10391 ts_tablet_manager.cc:1367] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Time spent bootstrapping tablet: real 0.221s user 0.029s sys 0.005s
I20240502 08:26:08.863806 10391 raft_consensus.cc:348] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:08.864014 10391 raft_consensus.cc:729] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f32202b9db457fb09872f6f51ea8ba, State: Initialized, Role: FOLLOWER
I20240502 08:26:08.864286 10391 consensus_queue.cc:260] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:08.864791 21942 raft_consensus.cc:1203] T 7b3f6a0b2561419ca6bd832fd65e3196 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0x7fabb43c2b93 google::LogMessage::SendToLog() at ??:0
I20240502 08:26:08.872130 10308 ts_tablet_manager.cc:1888] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:26:08.872550 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296ccf9be99545789bbb3ff4031d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:08.878680 21409 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "95f32202b9db457fb09872f6f51ea8ba" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:08.880919 23850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:26:08.882252 10308 tablet_copy_client.cc:783] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.888540 10391 ts_tablet_manager.cc:1398] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 08:26:08.889400 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 95f32202b9db457fb09872f6f51ea8ba-8a45b1da348e4bb89f021df103a0603f received from {username='slave'} at 127.8.68.200:42037
I20240502 08:26:08.889575 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 95f32202b9db457fb09872f6f51ea8ba-8a45b1da348e4bb89f021df103a0603f on tablet 8a45b1da348e4bb89f021df103a0603f with peer 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:08.895735 10308 tablet_copy_client.cc:647] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.916361 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:08.917202 10308 tablet_copy_client.cc:523] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:08.919059 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833): 4840 bytes on disk
I20240502 08:26:08.919778 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(76bb1e3e2b9d4e31bf93ac1ce08a7833) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:26:08.925664 10308 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:26:08.928309 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 59f5d97e6e0a4755be41699246907e55 from peer 95f32202b9db457fb09872f6f51ea8ba ({username='slave'} at 127.8.68.200:45605)
I20240502 08:26:08.928539 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 59f5d97e6e0a4755be41699246907e55 from peer 95f32202b9db457fb09872f6f51ea8ba at {username='slave'} at 127.8.68.200:45605: session id = 95f32202b9db457fb09872f6f51ea8ba-59f5d97e6e0a4755be41699246907e55
I20240502 08:26:08.936440 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:08.937363 21572 tablet_copy_source_session.cc:205] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:08.940536 10390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f5d97e6e0a4755be41699246907e55. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fabb43be7cc google::LogMessage::Flush() at ??:0
@ 0x7fabb43bff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:26:08.955909 10390 tablet_copy_client.cc:783] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:08.970535 10390 tablet_copy_client.cc:647] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:08.976301 10390 tablet_copy_client.cc:523] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fabb70ff742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:26:08.992200 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55288
I20240502 08:26:08.994218 23868 raft_consensus.cc:1203] T 8a45b1da348e4bb89f021df103a0603f P 95f32202b9db457fb09872f6f51ea8ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0x7fabb70cbf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:26:09.001545 23917 maintenance_manager.cc:419] P 95f32202b9db457fb09872f6f51ea8ba: Scheduling UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f): 4925 bytes on disk
I20240502 08:26:09.002346 23808 maintenance_manager.cc:643] P 95f32202b9db457fb09872f6f51ea8ba: UndoDeltaBlockGCOp(8a45b1da348e4bb89f021df103a0603f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
@ 0x7fabb70c5cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 08:26:09.004470 23868 consensus_queue.cc:237] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:09.006687 10462 tablet_replica.cc:330] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: stopping tablet replica
I20240502 08:26:09.006947 10462 raft_consensus.cc:2229] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:26:09.011710 23469 raft_consensus.cc:1261] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 95f32202b9db457fb09872f6f51ea8ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
@ 0x7fabc65188de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:26:09.015503 10462 raft_consensus.cc:2258] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:26:09.016832 10451 consensus_queue.cc:1035] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:26:09.017988 10462 ts_tablet_manager.cc:1875] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7fabc6530daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:26:09.023411 10235 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
@ 0x7fabc665b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:26:09.028271 10390 tablet_bootstrap.cc:492] T 59f5d97e6e0a4755be41699246907e55 P 95f32202b9db457fb09872f6f51ea8ba: Bootstrap starting.
I20240502 08:26:09.028378 23470 raft_consensus.cc:2932] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
@ 0x7fabc6693a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240502 08:26:09.036563 23806 consensus_peers.cc:485] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 95f32202b9db457fb09872f6f51ea8ba -> Peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Couldn't send request to peer 13e764b27ed449c09aee719cdb2597f8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:09.047725 21395 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
@ 0x7fabc6676981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fabc667670e std::_Function_handler<>::_M_invoke() at ??:0
I20240502 08:26:09.074460 21409 catalog_manager.cc:5484] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 11 to 12, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New cstate: current_term: 2 leader_uuid: "95f32202b9db457fb09872f6f51ea8ba" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:26:09.101495 10462 ts_tablet_manager.cc:1888] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:26:09.101647 10462 log.cc:1196] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 60677121510e47a5b51d1eb22f9aa238: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:26:09.102473 21398 catalog_manager.cc:4830] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa (table rebalance_test_table_2 [id=f04d6b2034c9494daf0dd7d90ed835a6]) successfully deleted
@ 0x7fabb70cfb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 08:26:09.128908 10308 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:09.130507 10308 tablet_bootstrap.cc:492] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:26:09.130905 10308 ts_tablet_manager.cc:1367] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.205s user 0.034s sys 0.008s
I20240502 08:26:09.132038 10308 raft_consensus.cc:348] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:09.132378 10308 raft_consensus.cc:729] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:26:09.132671 10308 consensus_queue.cc:260] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
@ 0x7fabb70d319c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 08:26:09.140558 10308 ts_tablet_manager.cc:1398] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:26:09.141335 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-296ccf9be99545789bbb3ff4031d3170 received from {username='slave'} at 127.8.68.196:57225
I20240502 08:26:09.141484 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-296ccf9be99545789bbb3ff4031d3170 on tablet 296ccf9be99545789bbb3ff4031d3170 with peer 0f655ab8f20d4ba482c3173f9594b98b
I20240502 08:26:09.145540 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170): 4586 bytes on disk
I20240502 08:26:09.146525 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(296ccf9be99545789bbb3ff4031d3170) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
@ 0x7fabb70d3019 std::_Function_handler<>::_M_invoke() at ??:0
I20240502 08:26:09.150331 21942 raft_consensus.cc:1203] T 296ccf9be99545789bbb3ff4031d3170 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240502 08:26:09.171092 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
@ 0x7fabb70ae01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabba4336db start_thread at ??:0
@ 0x7fabb327171f clone at ??:0
W20240502 08:26:09.393007 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:09.456743 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:09.471303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:09.493129 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:09.565213 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:26:09.568538 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:09.585904 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 08:26:09.658135 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:09.716881 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:09.737686 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:09.749331 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:09.784013 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:09.795322 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:09.796188 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:09.799772 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:09.873185 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:09.949843 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:10.066083 23404 connection.cc:535] client connection to 127.8.68.200:40881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.066135 21746 connection.cc:535] client connection to 127.8.68.200:40881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.066537 21485 connection.cc:535] client connection to 127.8.68.200:40881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.068197 21877 connection.cc:535] client connection to 127.8.68.200:40881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.069236 21616 connection.cc:535] client connection to 127.8.68.200:40881 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.071962 8783 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:26:10.067282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:40881 (local address 127.8.68.195:34437)
0502 08:26:10.067788 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0502 08:26:10.067810 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0502 08:26:10.067943 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:26:10.071372 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":365}
W20240502 08:26:10.072187 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:26:10.072309 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.072403 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.072484 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.075425 10555 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:26:10.067861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:40881 (local address 127.8.68.193:60651)
0502 08:26:10.068190 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0502 08:26:10.068207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 08:26:10.068322 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:26:10.074819 (+ 6497us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":209}
W20240502 08:26:10.075793 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.075896 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.076326 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.076457 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.076546 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.076625 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.101096 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:10.109094 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:10.151592 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.158145 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:10.199582 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:10.226429 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.284168 21662 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:10.319648 21682 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68"
dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55368
I20240502 08:26:10.319886 21682 raft_consensus.cc:595] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:26:10.409301 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:10.472172 21791 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:10.501461 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240502 08:26:10.528946 9286 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:10.529253 9286 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:10.530378 9286 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:10.532147 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:10.532717 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:10.532852 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:10.533569 9286 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:26:10.538794 9906 raft_consensus.cc:979] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238: : Instructing follower 13e764b27ed449c09aee719cdb2597f8 to start an election
I20240502 08:26:10.538961 9906 raft_consensus.cc:1067] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Signalling peer 13e764b27ed449c09aee719cdb2597f8 to start an election
I20240502 08:26:10.539842 21812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68"
dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
from {username='slave'} at 127.8.68.194:40215
I20240502 08:26:10.540836 21812 raft_consensus.cc:482] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:26:10.540936 21812 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:26:10.547111 21812 raft_consensus.cc:504] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:10.548518 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "60677121510e47a5b51d1eb22f9aa238"
I20240502 08:26:10.548764 21682 raft_consensus.cc:3032] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:26:10.548840 21682 raft_consensus.cc:729] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Running, Role: LEADER
I20240502 08:26:10.549108 21682 consensus_queue.cc:260] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:10.549453 21682 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:26:10.553416 21682 raft_consensus.cc:2454] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 2.
I20240502 08:26:10.554122 21749 leader_election.cc:302] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, 60677121510e47a5b51d1eb22f9aa238; no voters:
I20240502 08:26:10.554520 10648 raft_consensus.cc:2781] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:26:10.554677 10648 raft_consensus.cc:686] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 2 LEADER]: Becoming Leader. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Running, Role: LEADER
I20240502 08:26:10.554955 10648 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:10.557493 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599d5781f1e24ae9ad425a34d88c6b68" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "20b637fbec5444dd91b3af9cdda482f5"
I20240502 08:26:10.557722 23469 raft_consensus.cc:3037] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:26:10.562319 23469 raft_consensus.cc:2454] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e764b27ed449c09aee719cdb2597f8 in term 2.
I20240502 08:26:10.564427 21812 leader_election.cc:288] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 2 election: Requested vote from peers 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745), 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:26:10.564473 21410 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: term changed from 1 to 2, leader changed from 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) to 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:10.572170 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.580523 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:10.612576 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.620538 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:10.684342 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 08:26:10.705869 21682 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:10.716603 21812 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:10.717258 10605 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:10.748986 10605 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:10.759833 21812 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:26:10.773627 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.775372 21410 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 7 to 8, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
W20240502 08:26:10.782765 10605 consensus_queue.cc:1159] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "13e764b27ed449c09aee719cdb2597f8" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20240502 08:26:10.784473 21682 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:10.787173 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet b88c4a1a36424086aad89b7585c17960 with delete_type TABLET_DATA_TOMBSTONED (TS 0f655ab8f20d4ba482c3173f9594b98b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57198
I20240502 08:26:10.789804 21812 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:26:10.790653 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:10.791085 9906 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:10.793385 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:10.794857 9906 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:10.799329 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b88c4a1a36424086aad89b7585c17960 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:10.800267 21813 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:10.818092 21410 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 8 to 9, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } }
I20240502 08:26:10.826742 10713 tablet_replica.cc:330] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: stopping tablet replica
I20240502 08:26:10.827929 10713 raft_consensus.cc:2229] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:26:10.830796 10713 raft_consensus.cc:2258] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:26:10.832976 10713 ts_tablet_manager.cc:1875] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:26:10.846480 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:10.855124 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:10.866626 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:10.895622 10713 ts_tablet_manager.cc:1888] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:26:10.896232 10713 log.cc:1196] T b88c4a1a36424086aad89b7585c17960 P 0f655ab8f20d4ba482c3173f9594b98b: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/b88c4a1a36424086aad89b7585c17960
I20240502 08:26:10.897310 21398 catalog_manager.cc:4830] TS 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): tablet b88c4a1a36424086aad89b7585c17960 (table rebalance_test_table_5 [id=b6feb10e86734c6988fe85ecc1269f2d]) successfully deleted
W20240502 08:26:10.946524 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:11.051771 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:26:11.072098 21682 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:26:11.073071 10648 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:11.127200 23469 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:26:11.128846 10758 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:26:11.129567 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240502 08:26:11.167359 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:11.170425 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:11.207165 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:11.225584 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:11.226212 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:11.249899 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:11.328009 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:11.358129 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:11.743780 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:11.746450 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:11.786162 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 176 similar messages]
I20240502 08:26:11.804229 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:11.828523 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:11.867421 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:11.879760 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:11.911959 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:11.941296 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:11.953100 21813 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:11.955811 23469 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:26:11.956576 10752 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:11.960222 10854 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:11.963495 23469 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:11.970295 21404 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 7 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:26:11.991180 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:12.060314 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:26:12.072724 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:12.074326 21662 tablet_service.cc:1514] Processing DeleteTablet for tablet 599d5781f1e24ae9ad425a34d88c6b68 with delete_type TABLET_DATA_TOMBSTONED (TS 60677121510e47a5b51d1eb22f9aa238 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55288
I20240502 08:26:12.075449 21813 consensus_queue.cc:237] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:26:12.077760 21682 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:12.078711 23469 raft_consensus.cc:1261] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:26:12.079717 10854 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:26:12.086946 10854 consensus_queue.cc:1035] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:12.089753 21682 raft_consensus.cc:858] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
W20240502 08:26:12.089880 21682 raft_consensus.cc:1565] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 NON_PARTICIPANT]: Could not prepare op 'term: 2 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:26:12.090382 21682 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 NON_PARTICIPANT]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:12.093654 10854 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:26:12.099422 23469 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:26:12.102653 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 599d5781f1e24ae9ad425a34d88c6b68 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:12.104736 21410 catalog_manager.cc:5484] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 9 to 10, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New cstate: current_term: 2 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
W20240502 08:26:12.124284 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:12.153069 21683 raft_consensus.cc:1467] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 08:26:12.153757 21683 raft_consensus.cc:2932] T 599d5781f1e24ae9ad425a34d88c6b68 P 60677121510e47a5b51d1eb22f9aa238 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
W20240502 08:26:12.162040 21398 catalog_manager.cc:4804] TS 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): delete failed for tablet 599d5781f1e24ae9ad425a34d88c6b68 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 7 but the committed config has opid_index of 10
W20240502 08:26:12.166889 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.208822 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.212452 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.212615 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:12.232933 23924 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskivhcCF/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 08:20:17.522848 23927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:17.523362 23927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:17.536435 23927 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 08:20:20.461786 24383 meta_cache.cc:1330] Time spent looking up entry by ID: real 0.087s user 0.000s sys 0.000s
I20240502 08:20:21.203879 23927 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:20:22.965754 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.965983 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.966100 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.966213 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.966333 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.966514 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.966634 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.966778 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.966907 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.967052 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.967347 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.967497 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.968012 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.968319 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
I20240502 08:20:23.310482 23927 rebalancer_tool.cc:1084] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '95f32202b9db457fb09872f6f51ea8ba' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:20:23.310932 23927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:25.005823 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.006067 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.006423 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.006597 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.006732 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.006875 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.006999 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.007141 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.007311 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.007459 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.007571 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.007680 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.007791 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.007918 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:25.142028 23927 rebalancer_tool.cc:1074] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:25.574156 23927 rebalancer_tool.cc:1074] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:26.052500 23927 rebalancer_tool.cc:1074] tablet cec56facd7f740aeb42bc7c32495dae3: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:26.765817 23927 rebalancer_tool.cc:1074] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:27.373943 23927 rebalancer_tool.cc:1074] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:27.905460 23927 rebalancer_tool.cc:1074] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: 'cfff665685a14d8796aece26addab59c' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:28.073035 23927 rebalancer_tool.cc:1084] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move ignored: Illegal state: tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: replica fd5e97ae7e344e4687b15ca0cde8d0ba is already present
I20240502 08:20:28.073220 23927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:30.303864 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.304280 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.304383 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.304570 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:30.304646 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:30.304737 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.304876 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:30.305119 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.305208 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.305308 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:30.436360 23927 rebalancer_tool.cc:1074] tablet 12c17796e5d04014aafdac46984f4ba9: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:20:30.819453 23927 rebalancer_tool.cc:1074] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:31.448096 23927 rebalancer_tool.cc:1074] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:32.360127 23927 rebalancer_tool.cc:1074] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:32.629820 23927 rebalancer_tool.cc:1074] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:32.831758 23927 rebalancer_tool.cc:1084] tablet 203e7914da3f4494b239c6603513caee: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move ignored: Illegal state: tablet 203e7914da3f4494b239c6603513caee: replica 20b637fbec5444dd91b3af9cdda482f5 is already present
I20240502 08:20:32.831930 23927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:34.888686 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:34.889158 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:34.889282 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server fd5e97ae7e344e4687b15ca0cde8d0ba to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:34.889389 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:34.889568 23927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:35.022125 23927 rebalancer_tool.cc:1074] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:35.110941 23927 rebalancer_tool.cc:1074] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:44.238230 23927 rebalancer_tool.cc:1123] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move is abandoned: Not found: Server with UUID cfff665685a14d8796aece26addab59c not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:51.458513 23927 rebalancer_tool.cc:1136] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:54.138429 23927 rebalancer_tool.cc:1136] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:58.503278 23927 rebalancer_tool.cc:1136] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: Incomplete: tablet 80b3ba1d834a4314b63f336675845441, TS 60677121510e47a5b51d1eb22f9aa238 -> TS fd5e97ae7e344e4687b15ca0cde8d0ba move failed, target replica disappeared
I20240502 08:21:01.141480 23927 rebalancer_tool.cc:1074] tablet e0241228bd204ca2a8ad2c57b6eeb543: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:01.216034 23927 rebalancer_tool.cc:1136] tablet e0241228bd204ca2a8ad2c57b6eeb543: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:01.258747 23927 rebalancer_tool.cc:1136] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:01.384512 23927 rebalancer_tool.cc:1136] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:01.465845 23927 rebalancer_tool.cc:1136] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:01.546947 23927 rebalancer_tool.cc:1136] tablet 12c17796e5d04014aafdac46984f4ba9: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:01.620218 23927 rebalancer_tool.cc:1136] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:01.719374 23927 rebalancer_tool.cc:1136] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:01.916699 23927 rebalancer_tool.cc:1136] tablet cec56facd7f740aeb42bc7c32495dae3: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:02.335701 23927 rebalancer_tool.cc:1136] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:02.431370 23927 rebalancer_tool.cc:1136] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: 'cfff665685a14d8796aece26addab59c' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:02.467964 23927 rebalancer_tool.cc:1074] tablet 9c0a886b3ba04737a6988c3b58c9698e: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:21:02.601070 23927 rebalancer_tool.cc:1074] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:02.768414 23927 rebalancer_tool.cc:1074] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:02.842473 23927 rebalancer_tool.cc:1074] tablet 852613f14b324b38b5eecc29d4c602cd: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:21:04.834245 23927 rebalancer_tool.cc:1123] tablet 852613f14b324b38b5eecc29d4c602cd: 'cfff665685a14d8796aece26addab59c' -> '' move is abandoned: Not found: Server with UUID cfff665685a14d8796aece26addab59c not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:21:04.910734 23927 rebalancer_tool.cc:1136] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:05.073490 23927 rebalancer_tool.cc:1136] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:05.172008 23927 rebalancer_tool.cc:1136] tablet 9c0a886b3ba04737a6988c3b58c9698e: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:05.620975 23927 rebalancer_tool.cc:1074] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:05.836951 23927 rebalancer_tool.cc:1074] tablet 203e7914da3f4494b239c6603513caee: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:05.925897 23927 rebalancer_tool.cc:1074] tablet dc49c050280346db88d1323e30513c80: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:21:06.010843 23927 rebalancer_tool.cc:1136] tablet dc49c050280346db88d1323e30513c80: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:06.055594 23927 rebalancer_tool.cc:1136] tablet 203e7914da3f4494b239c6603513caee: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:06.115157 23927 rebalancer_tool.cc:1136] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:06.246922 23927 rebalancer_tool.cc:1074] tablet b88c4a1a36424086aad89b7585c17960: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:21:06.343740 23927 rebalancer_tool.cc:1074] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:06.449921 23927 rebalancer_tool.cc:1074] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:06.517513 23927 rebalancer_tool.cc:1136] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:06.687289 23927 rebalancer_tool.cc:1136] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
W20240502 08:21:08.793991 31920 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:07.872736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:41299 (local address 127.0.0.1:34044)
0502 08:21:07.892565 (+ 19829us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:07.892585 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:07.921963 (+ 29378us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x61300025f330. stack: 00007ff0d43fb756 00007ff0d43fb510 00007ff0d342e48d 0000000000955804 00000000009557cd 000000000095519a 00007ff0d43f119d 00007ff0d43f0dbe 00007ff0e78a962d 00007ff0e78a95cc 00007ff0e78759bb 00007ff0dafcade6 00007ff0dafc965e 00007ff0db0524cd 00007ff0db050612 00007ff0db0841bf
0502 08:21:07.922017 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:08.792579 (+870562us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19721,"spinlock_wait_cycles":56257792,"thread_start_us":19980,"threads_started":1}
W20240502 08:21:08.823143 31920 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:08.795528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:41299 (local address 127.0.0.1:34064)
0502 08:21:08.795773 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:08.795794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:08.795909 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:08.822922 (+ 27013us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133}
I20240502 08:21:08.966564 23927 rebalancer_tool.cc:1074] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:21:08.993894 23927 rebalancer_tool.cc:1074] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:21:09.049250 23927 rebalancer_tool.cc:1136] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:09.228561 23927 rebalancer_tool.cc:1136] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
W20240502 08:22:01.738580 7031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 51906.4us
W20240502 08:22:01.760494 7029 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 79769.3us
W20240502 08:22:19.430210 9598 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.000s sys 0.000s
W20240502 08:22:19.737641 9487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 79003.5us
W20240502 08:25:41.286146 6622 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.Quiesce -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 60710.6us
I20240502 08:26:11.155514 23927 rebalancer_tool.cc:1123] tablet b88c4a1a36424086aad89b7585c17960: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move is abandoned: Not found: Server with UUID 0f655ab8f20d4ba482c3173f9594b98b not a member of the config. RaftConfig: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:11.155647 23927 rebalancer_tool.cc:748] detected a staleness period of 300.443s
Incomplete: stalled with no progress for more than 300.000s seconds, aborting
W20240502 08:26:12.269134 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:12.303066 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.307344 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:12.315636 23923 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskivhcCF/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 08:20:17.537629 23926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:17.538046 23926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:17.563347 23926 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:20:21.139326 23926 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:20:22.356389 24548 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 62004.5us
W20240502 08:20:23.000181 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:23.000532 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:23.001960 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:23.002421 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:23.002626 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:23.002885 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:23.003031 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:23.003178 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:23.003314 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:23.003464 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:23.003594 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:23.003754 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:23.003844 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:23.003930 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:23.004024 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:23.173992 23926 rebalancer_tool.cc:1084] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move ignored: Illegal state: tablet 3898bfdb9c3d4a05af78f79fd2de61e2: replica 20b637fbec5444dd91b3af9cdda482f5 is already present
I20240502 08:20:23.174252 23926 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:25.040376 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.040766 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.040910 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.041004 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.041132 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.043387 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.043551 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.043857 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.043998 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.044129 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.044245 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.044337 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.044435 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.044528 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:25.164619 23926 rebalancer_tool.cc:1074] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:25.518643 23926 rebalancer_tool.cc:1074] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:26.105870 23926 rebalancer_tool.cc:1074] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:26.910300 23926 rebalancer_tool.cc:1074] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:27.393309 23926 rebalancer_tool.cc:1084] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:20:27.393473 23926 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:30.276193 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.277118 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:30.277395 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.277588 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.277762 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.277899 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:30.278070 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.278270 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:30.278414 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:30.278857 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:30.279031 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:20:30.848522 23926 rebalancer_tool.cc:1074] tablet 203e7914da3f4494b239c6603513caee: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:30.892554 23926 rebalancer_tool.cc:1084] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move ignored: Illegal state: tablet c4862ea2f67c43a28689b464d8ec96e6: replica 95f32202b9db457fb09872f6f51ea8ba is already present
I20240502 08:20:30.892751 23926 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:32.947021 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:32.947506 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:32.947825 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:32.948118 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:32.948256 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:32.948387 23926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:33.163147 23926 rebalancer_tool.cc:1074] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:33.687438 23926 rebalancer_tool.cc:1074] tablet 2bfc3eaed583458b8381d4e7b4d36aef: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:33.994972 23926 rebalancer_tool.cc:1074] tablet 9c0a886b3ba04737a6988c3b58c9698e: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:34.360904 23926 rebalancer_tool.cc:1074] tablet b88c4a1a36424086aad89b7585c17960: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:34.849910 23926 rebalancer_tool.cc:1074] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:50.548136 23926 rebalancer_tool.cc:1136] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:20:50.640070 23926 rebalancer_tool.cc:1136] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:52.756630 23926 rebalancer_tool.cc:1123] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '' move is abandoned: Not found: Server with UUID cfff665685a14d8796aece26addab59c not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:52.890799 23926 rebalancer_tool.cc:1136] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:53.029009 23926 rebalancer_tool.cc:1074] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:20:53.281033 23926 rebalancer_tool.cc:1074] tablet a20b2fd9f1ce488ba7793ed7ac268c16: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:55.538354 23926 rebalancer_tool.cc:1136] tablet b88c4a1a36424086aad89b7585c17960: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:57.889792 23926 rebalancer_tool.cc:1136] tablet 2bfc3eaed583458b8381d4e7b4d36aef: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:00.650884 23926 rebalancer_tool.cc:1123] tablet a20b2fd9f1ce488ba7793ed7ac268c16: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move is abandoned: Not found: Server with UUID 0f655ab8f20d4ba482c3173f9594b98b not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:21:00.973021 23926 rebalancer_tool.cc:1136] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:01.046294 23926 rebalancer_tool.cc:1136] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:01.092797 23926 rebalancer_tool.cc:1136] tablet 203e7914da3f4494b239c6603513caee: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:01.169742 23926 rebalancer_tool.cc:1074] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:01.313267 23926 rebalancer_tool.cc:1136] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:01.375824 23926 rebalancer_tool.cc:1136] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:01.514740 23926 rebalancer_tool.cc:1136] tablet 9c0a886b3ba04737a6988c3b58c9698e: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:01.576817 23926 rebalancer_tool.cc:1074] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:01.797490 23926 rebalancer_tool.cc:1074] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:01.849988 23926 rebalancer_tool.cc:1074] tablet 852613f14b324b38b5eecc29d4c602cd: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:21:04.446238 23926 rebalancer_tool.cc:1136] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
W20240502 08:21:07.021857 31590 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.Quiesce -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 54565.2us
W20240502 08:21:07.948813 31750 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:07.238652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.199:33397 (local address 127.0.0.1:55492)
0502 08:21:07.238988 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:07.239010 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:07.239123 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:07.947301 (+708178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":164}
I20240502 08:21:08.139478 23926 rebalancer_tool.cc:1074] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:08.329741 23926 rebalancer_tool.cc:1136] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:08.464771 23926 rebalancer_tool.cc:1136] tablet 852613f14b324b38b5eecc29d4c602cd: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:09.684471 23926 rebalancer_tool.cc:1074] tablet e0241228bd204ca2a8ad2c57b6eeb543: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:09.851202 23926 rebalancer_tool.cc:1074] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:21:10.015990 23926 rebalancer_tool.cc:1136] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:10.099812 23926 rebalancer_tool.cc:1136] tablet e0241228bd204ca2a8ad2c57b6eeb543: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:11.448488 23926 rebalancer_tool.cc:1074] tablet cec56facd7f740aeb42bc7c32495dae3: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:11.516223 23926 rebalancer_tool.cc:1074] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:21:11.889827 23926 rebalancer_tool.cc:1136] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:12.042557 23926 rebalancer_tool.cc:1136] tablet cec56facd7f740aeb42bc7c32495dae3: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:13.660768 23926 rebalancer_tool.cc:1074] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:21:13.742537 23926 rebalancer_tool.cc:1074] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:13.810387 23926 rebalancer_tool.cc:1136] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:21:14.025312 23926 rebalancer_tool.cc:1136] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:15.603678 23926 rebalancer_tool.cc:1074] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:15.633292 23926 rebalancer_tool.cc:1136] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
W20240502 08:21:33.562351 3066 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.68.194:36745, user_credentials={real_user=slave}} blocked reactor thread for 46349.1us
W20240502 08:22:45.843133 13410 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 44262.9us
W20240502 08:25:54.396939 8395 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 50529.5us
W20240502 08:25:58.114552 9031 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.000s sys 0.000s
I20240502 08:26:09.160910 23926 rebalancer_tool.cc:1136] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
Illegal state: tablet server 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): unacceptable health status UNAVAILABLE
I20240502 08:26:12.327349 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:12.358500 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2de41d6c6de0400397645c05a0fa17de: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:12.365175 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:12.387221 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:12.399837 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:12.431092 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:12.437899 9286 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:12.438138 9286 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:12.438915 9286 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:12.447885 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:12.448251 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240502 08:26:12.459331 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:12.459508 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:12.460276 9286 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:26:12.500303 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:12.533854 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.542317 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:12.688722 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:12.692597 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:12.710227 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:12.710481 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:12.741320 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:12.775559 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:12.786188 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:12.850050 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:13.062462 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.087306 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:13.125164 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:13.134202 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:13.187481 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:26:13.202808 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.254722 23921 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskivhcCF/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 08:20:17.338888 23928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:17.339422 23928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:17.353245 23928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:20:20.970113 23928 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:20:22.830196 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.830425 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.830730 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.830904 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.831036 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.831152 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.831296 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.831410 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.831516 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.831651 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.831743 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.831851 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.831941 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.832052 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:23.132562 23928 rebalancer_tool.cc:1074] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:23.476907 23928 rebalancer_tool.cc:1074] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:23.797451 23928 rebalancer_tool.cc:1074] tablet 852613f14b324b38b5eecc29d4c602cd: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:25.018869 23928 rebalancer_tool.cc:1074] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:25.372371 23928 rebalancer_tool.cc:1074] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:25.759752 23928 rebalancer_tool.cc:1074] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:26.426692 23928 rebalancer_tool.cc:1074] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:26.839772 23928 rebalancer_tool.cc:1074] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:27.505951 23928 rebalancer_tool.cc:1074] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:27.845100 23928 rebalancer_tool.cc:1074] tablet e0241228bd204ca2a8ad2c57b6eeb543: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:28.816851 23928 rebalancer_tool.cc:1074] tablet a20b2fd9f1ce488ba7793ed7ac268c16: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:29.756515 23928 rebalancer_tool.cc:1074] tablet dc49c050280346db88d1323e30513c80: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:30.382277 23928 rebalancer_tool.cc:1074] tablet 12c17796e5d04014aafdac46984f4ba9: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:30.725778 23928 rebalancer_tool.cc:1074] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:31.053179 23928 rebalancer_tool.cc:1074] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '0f655ab8f20d4ba482c3173f9594b98b' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:31.753118 23928 rebalancer_tool.cc:1074] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:32.329870 23928 rebalancer_tool.cc:1074] tablet 59f5d97e6e0a4755be41699246907e55: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:32.917775 23928 rebalancer_tool.cc:1074] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:33.033685 23928 rebalancer_tool.cc:1084] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move ignored: Illegal state: tablet 182a6f2df91242cfa10f05ef788a19cb: replica fd5e97ae7e344e4687b15ca0cde8d0ba is already present
I20240502 08:20:33.033936 23928 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:34.367412 26515 meta_cache.cc:1041] Time spent processing tablet locations response: real 0.064s user 0.000s sys 0.000s
W20240502 08:20:34.368554 26515 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 65376.9us
W20240502 08:20:34.520350 26515 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 51530us
W20240502 08:20:35.090622 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server fd5e97ae7e344e4687b15ca0cde8d0ba to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.090981 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 95f32202b9db457fb09872f6f51ea8ba to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.091147 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:35.091317 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:35.091434 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:35.091614 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 4b9bfd2541144dad86507451343640d3 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:35.091795 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:35.091938 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:35.092767 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.092949 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:35.093089 23928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:35.147115 23928 rebalancer_tool.cc:1074] tablet 182a6f2df91242cfa10f05ef788a19cb: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:35.211416 23928 rebalancer_tool.cc:1074] tablet 203e7914da3f4494b239c6603513caee: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:44.502503 23928 rebalancer_tool.cc:1123] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move is abandoned: Not found: Server with UUID cfff665685a14d8796aece26addab59c not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } }
I20240502 08:20:54.453608 23928 rebalancer_tool.cc:1136] tablet 852613f14b324b38b5eecc29d4c602cd: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:20:56.275096 23928 rebalancer_tool.cc:1123] tablet ebf79c69fd614284ba45fd052cac7ebf: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move is abandoned: Not found: Server with UUID 60677121510e47a5b51d1eb22f9aa238 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:20:56.388821 23928 rebalancer_tool.cc:1136] tablet 296ccf9be99545789bbb3ff4031d3170: '0f655ab8f20d4ba482c3173f9594b98b' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:20:58.495371 23928 rebalancer_tool.cc:1136] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '0f655ab8f20d4ba482c3173f9594b98b' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:58.611407 23928 rebalancer_tool.cc:1136] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:21:00.931874 23928 rebalancer_tool.cc:1136] tablet 203e7914da3f4494b239c6603513caee: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:01.014094 23928 rebalancer_tool.cc:1136] tablet e0241228bd204ca2a8ad2c57b6eeb543: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:01.097497 23928 rebalancer_tool.cc:1136] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:03.085552 23928 rebalancer_tool.cc:1136] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:03.150478 23928 rebalancer_tool.cc:1074] tablet b88c4a1a36424086aad89b7585c17960: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
W20240502 08:21:04.084405 31193 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.001s sys 0.000s
I20240502 08:21:04.858601 23928 rebalancer_tool.cc:1136] tablet 182a6f2df91242cfa10f05ef788a19cb: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:04.947845 23928 rebalancer_tool.cc:1136] tablet 59f5d97e6e0a4755be41699246907e55: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:21:05.042498 23928 rebalancer_tool.cc:1136] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:05.301262 23928 rebalancer_tool.cc:1136] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:05.566509 23928 rebalancer_tool.cc:1136] tablet dc49c050280346db88d1323e30513c80: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:21:05.806054 23928 rebalancer_tool.cc:1136] tablet 12c17796e5d04014aafdac46984f4ba9: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:05.914933 23928 rebalancer_tool.cc:1136] tablet d62f6eec9ee04f93a3738a83f790292c: '60677121510e47a5b51d1eb22f9aa238' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:05.994186 23928 rebalancer_tool.cc:1136] tablet a20b2fd9f1ce488ba7793ed7ac268c16: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:21:06.042403 23928 rebalancer_tool.cc:1136] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '13e764b27ed449c09aee719cdb2597f8' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:06.150637 23928 rebalancer_tool.cc:1136] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:06.290792 23928 rebalancer_tool.cc:1074] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:06.734858 23928 rebalancer_tool.cc:1074] tablet d4d933eacd944d84b39f0cb9c45641f8: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:21:06.834800 23928 rebalancer_tool.cc:1074] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:21:07.266460 23928 rebalancer_tool.cc:1136] tablet d4d933eacd944d84b39f0cb9c45641f8: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:21:07.411069 23928 rebalancer_tool.cc:1136] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:07.529983 23928 rebalancer_tool.cc:1136] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
W20240502 08:21:08.826293 32020 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:08.584855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:41299 (local address 127.0.0.1:34052)
0502 08:21:08.603386 (+ 18531us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:08.603414 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:08.603542 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:08.824953 (+221411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15106,"thread_start_us":120862,"threads_started":1}
W20240502 08:21:31.413131 2863 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 55157.3us
W20240502 08:22:43.255318 12911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 47821us
W20240502 08:24:50.986388 32221 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 66801.9us
W20240502 08:24:52.312737 32537 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 44647.7us
W20240502 08:26:10.068785 10571 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:26:09.716674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:40881 (local address 127.0.0.1:40620)
0502 08:26:09.725526 (+ 8852us) negotiation.cc:107] Waiting for socket to connect
0502 08:26:09.725554 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0502 08:26:09.725732 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:26:10.068553 (+342821us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":8489}
I20240502 08:26:11.040879 23928 rebalancer_tool.cc:1136] tablet b88c4a1a36424086aad89b7585c17960: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
Illegal state: tablet server 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): unacceptable health status UNAVAILABLE
W20240502 08:26:13.267421 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:13.418247 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.442129 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:13.467170 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.512533 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:13.516441 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:13.615554 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:26:13.649683 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.666963 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:13.690181 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:13.693284 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:13.694345 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:13.734167 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:13.783908 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:13.804980 21682 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:26:13.815788 23469 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:26:13.816394 10792 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:26:13.819538 11066 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:13.820307 23469 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:13.826294 21409 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 7 to 8, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:26:13.859784 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with delete_type TABLET_DATA_TOMBSTONED (TS cfff665685a14d8796aece26addab59c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:58814
I20240502 08:26:13.872512 11087 tablet_replica.cc:330] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: stopping tablet replica
I20240502 08:26:13.878135 11087 raft_consensus.cc:2229] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:26:13.878537 11087 raft_consensus.cc:2258] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:26:13.879835 11087 ts_tablet_manager.cc:1875] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:26:13.883541 21682 consensus_queue.cc:237] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
W20240502 08:26:13.885802 21619 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395): Couldn't send request to peer 0f655ab8f20d4ba482c3173f9594b98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d96a7174eb5442f2b0cc0cafc3cc4c80. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:13.885826 23469 raft_consensus.cc:1261] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:26:13.886422 11066 consensus_queue.cc:1035] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:26:13.889771 11066 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:13.890272 23469 raft_consensus.cc:2932] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:13.896739 21410 catalog_manager.cc:5484] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 20b637fbec5444dd91b3af9cdda482f5 reported cstate change: config changed from index 8 to 9, VOTER 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:26:13.898382 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:26:13.905913 11087 ts_tablet_manager.cc:1888] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:26:13.906065 11087 log.cc:1196] T d96a7174eb5442f2b0cc0cafc3cc4c80 P cfff665685a14d8796aece26addab59c: Deleting WAL directory at /tmp/dist-test-taskivhcCF/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714637795628858-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/d96a7174eb5442f2b0cc0cafc3cc4c80
I20240502 08:26:13.906948 21398 catalog_manager.cc:4830] TS cfff665685a14d8796aece26addab59c (127.8.68.193:46845): tablet d96a7174eb5442f2b0cc0cafc3cc4c80 (table rebalance_test_table_3 [id=32bbe3ff8a894fef891c38475cbd42d8]) successfully deleted
I20240502 08:26:14.055887 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:14.101256 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:14.133846 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:14.154095 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:14.154809 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:14.162920 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:14.181308 9286 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:14.181514 9286 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:14.182214 9286 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:14.184420 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:14.185449 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:26:14.185570 23406 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:14.186092 9286 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:26:14.197039 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:14.208607 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:14.225571 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:14.247205 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:14.318928 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:14.336284 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:14.354183 11154 ts_tablet_manager.cc:902] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:26:14.363902 11154 tablet_copy_client.cc:308] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:26:14.374722 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 0f655ab8f20d4ba482c3173f9594b98b ({username='slave'} at 127.8.68.196:43563)
I20240502 08:26:14.374956 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 from peer 0f655ab8f20d4ba482c3173f9594b98b at {username='slave'} at 127.8.68.196:43563: session id = 0f655ab8f20d4ba482c3173f9594b98b-d96a7174eb5442f2b0cc0cafc3cc4c80
I20240502 08:26:14.378829 21703 tablet_copy_source_session.cc:205] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:26:14.385861 11154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96a7174eb5442f2b0cc0cafc3cc4c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:26:14.395113 11154 tablet_copy_client.cc:783] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 5 data blocks...
I20240502 08:26:14.404096 11154 tablet_copy_client.cc:647] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:26:14.405951 11154 tablet_copy_client.cc:523] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:26:14.409828 11154 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap starting.
I20240502 08:26:14.429951 11154 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:26:14.430404 11154 tablet_bootstrap.cc:492] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Bootstrap complete.
I20240502 08:26:14.430696 11154 ts_tablet_manager.cc:1367] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240502 08:26:14.431452 11154 raft_consensus.cc:348] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:14.431699 11154 raft_consensus.cc:729] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f655ab8f20d4ba482c3173f9594b98b, State: Initialized, Role: FOLLOWER
I20240502 08:26:14.431995 11154 consensus_queue.cc:260] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:26:14.432677 11154 ts_tablet_manager.cc:1398] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:26:14.433918 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-d96a7174eb5442f2b0cc0cafc3cc4c80 received from {username='slave'} at 127.8.68.196:43563
I20240502 08:26:14.434059 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 0f655ab8f20d4ba482c3173f9594b98b-d96a7174eb5442f2b0cc0cafc3cc4c80 on tablet d96a7174eb5442f2b0cc0cafc3cc4c80 with peer 0f655ab8f20d4ba482c3173f9594b98b
I20240502 08:26:14.463567 21550 consensus_queue.cc:237] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "4b9bfd2541144dad86507451343640d3" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 41299 } } peers { permanent_uuid: "fd5e97ae7e344e4687b15ca0cde8d0ba" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 33397 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:26:14.468865 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7021 bytes on disk
W20240502 08:26:14.470983 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:14.472147 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:26:14.472759 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:26:14.474931 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":359,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":125,"thread_start_us":259,"threads_started":1}
W20240502 08:26:14.526988 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:14.564181 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:14.612756 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 08:26:14.613759 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:14.640724 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:14.668874 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:14.679455 21990 maintenance_manager.cc:419] P 0f655ab8f20d4ba482c3173f9594b98b: Scheduling UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80): 4925 bytes on disk
I20240502 08:26:14.680243 21881 maintenance_manager.cc:643] P 0f655ab8f20d4ba482c3173f9594b98b: UndoDeltaBlockGCOp(d96a7174eb5442f2b0cc0cafc3cc4c80) complete. Timing: real 0.000s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
W20240502 08:26:14.724804 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:14.748775 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:14.749444 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:14.784296 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:14.787067 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:14.799959 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:14.801383 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:14.854409 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:14.868726 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:14.891269 21944 raft_consensus.cc:1203] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 08:26:14.903954 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:14.904806 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:14.913107 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:14.958447 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:14.999913 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:15.105365 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:15.107218 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:15.111830 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:15.114485 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:15.127316 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:15.180792 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:15.261782 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:15.353751 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:15.355000 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:15.372951 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:15.406085 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:15.438267 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:15.460434 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:15.481081 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:15.526469 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:15.636451 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:15.659909 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:15.660012 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:15.660044 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:15.660943 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:15.768436 9286 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:15.768654 9286 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:15.769354 9286 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:15.770565 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:15.771027 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:15.771122 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:15.771381 9286 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:26:15.805619 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:26:15.825698 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:26:15.945547 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:16.071463 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:16.082396 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:16.096159 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:16.121510 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:16.122349 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:16.320740 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:16.368629 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:16.372956 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:16.373867 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:16.387616 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:16.395514 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:16.412434 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:16.495146 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2de41d6c6de0400397645c05a0fa17de: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:16.570358 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:16.585139 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:16.593966 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:16.607684 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:16.623383 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:16.655212 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:16.735318 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:16.879845 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:16.889632 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:16.906922 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:16.912910 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:16.940220 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:26:16.948319 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:17.016808 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:17.069609 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:17.100416 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:17.107697 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:17.125733 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:17.152179 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:17.190075 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:17.242848 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:26:17.260885 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:17.290773 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:17.292083 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:17.393868 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:17.396272 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:17.413144 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:17.433573 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:17.447779 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:17.448163 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:17.484234 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:17.504402 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:17.526856 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:17.617527 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:17.652807 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:17.684113 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:17.685428 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 08:26:17.697829 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:17.705094 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:17.719564 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:17.731604 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:17.752696 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:17.870188 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:17.965416 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.002708 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:18.006651 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.007817 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:18.016255 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:18.201604 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:18.230453 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.234982 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:18.239023 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:18.251375 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:18.254462 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:18.262926 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.719408 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.722463 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:18.731009 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:18.731459 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:18.756004 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:18.765377 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:18.818500 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:19.009699 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:19.023772 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:19.041252 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:19.045886 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:19.046991 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:19.236606 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:26:19.315196 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:19.329761 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:19.356169 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:19.362846 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:19.363147 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:19.377626 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:19.456102 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:19.491326 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:19.496253 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.497591 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:19.498477 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:19.513396 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.585372 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:26:19.593698 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:19.594885 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:19.596098 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:19.605255 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:19.636193 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:19.640707 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.660867 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:19.662106 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:26:19.672245 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.691385 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:19.731215 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.776898 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:19.963289 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:19.970517 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:20.041157 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:20.041417 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:20.050899 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:26:20.059500 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:20.065289 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:20.070340 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:20.073818 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:20.146272 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:26:20.290805 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:20.298619 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:20.313709 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:20.314363 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:20.326273 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:20.329967 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:20.477361 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:26:20.539472 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:20.551473 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:20.560310 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:20.561647 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:20.605396 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:20.818943 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:26:20.855335 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:20.868813 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:20.869470 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:20.870055 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:20.885928 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:20.909039 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:20.914815 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:20.983705 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:26:20.983973 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:21.320458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:21.326726 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:21.341282 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:21.363552 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:21.375366 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:21.377979 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:21.427683 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:26:21.615694 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:21.626313 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:21.631363 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:21.634373 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:21.640125 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:21.658578 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:21.754492 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:21.841408 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:21.849330 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:21.849639 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:26:21.901602 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:21.901736 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:21.903882 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:21.934880 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:21.942481 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:21.947736 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:21.979416 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:21.997947 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:26:22.040469 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:22.045630 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:22.145675 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:22.169890 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:22.177920 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:22.184577 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:22.191758 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:22.236418 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:22.271181 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:22.276520 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:22.348623 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:22.361617 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:22.371589 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:22.381321 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:22.494882 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:22.591344 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:22.609082 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:22.614487 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:22.614826 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:22.616453 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:22.637007 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:22.767959 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:26:22.769002 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:22.864874 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:22.868768 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:22.886102 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:22.893836 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:22.930186 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:22.938663 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.129102 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.134685 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:23.152659 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:23.153486 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:23.159711 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:23.168272 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:23.281577 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:23.287407 11632 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:23.287650 11632 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:23.288800 11632 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:23.292346 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:23.293689 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:23.293807 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:23.294330 11632 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:26:23.389254 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:23.390411 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.392292 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:23.412364 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:26:23.414566 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.433142 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:23.495337 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:23.704493 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 08:26:23.705780 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
I20240502 08:26:23.875506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.885669 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:23.894234 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:23.911778 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:23.916500 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:24.002506 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:24.049479 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:24.120808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:24.148322 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:24.150385 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:24.152128 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:24.154894 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:24.294734 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:24.363333 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:24.373580 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:24.384415 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:24.387393 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:24.404059 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:24.411365 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:24.418112 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:24.418761 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:24.422180 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:24.525040 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:24.563097 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:24.604091 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:24.660622 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:24.661999 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:24.672416 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:24.675331 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:24.676301 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:24.694569 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:24.706398 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2de41d6c6de0400397645c05a0fa17de: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:24.742456 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:26:24.748191 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:26:24.751561 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:24.788163 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:24.814534 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:24.857810 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:24.889087 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:24.919185 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:24.931167 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:24.966885 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:25.110595 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.123597 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:25.127183 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:25.137722 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:25.142923 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:25.382810 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:25.396705 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.405047 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:25.406188 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:25.416432 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.435233 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:25.487077 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:25.547729 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:25.642230 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.651170 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:25.651543 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:25.656316 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:25.658917 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:25.761991 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:25.904350 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.911657 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:25.918654 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:25.935777 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:25.940985 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:25.956812 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:25.964803 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:26:26.001917 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:26.254840 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:26:26.329670 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:26.410444 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:26.412462 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:26.413625 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:26.423555 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:26.450043 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:26.487883 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:26:26.521725 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:26.638857 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:26.660050 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:26.674396 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:26.679862 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:26.683138 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:26.862793 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:26.870887 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:26.871167 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:26.877825 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:26.916064 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:26.932008 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:26.932614 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:26.937994 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:26.942430 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:26.969836 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:27.037058 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:27.077522 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:27.102787 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:27.135216 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:27.159365 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:27.168033 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:27.189008 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:27.195317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:27.196977 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:27.211205 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:27.211319 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:27.262838 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:27.341306 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:26:27.343295 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:27.428540 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:27.480837 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:27.513844 11632 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:27.514056 11632 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:27.514808 11632 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:27.516350 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:27.517077 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:27.517187 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:27.517429 11632 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:26:27.581871 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:27.628926 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:27.643029 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:27.645885 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:27.656364 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:27.672068 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:27.709707 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:27.774581 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:26:27.775501 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:27.908324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:27.912099 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:27.935592 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:27.941416 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:27.944165 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:27.957391 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:27.962288 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:28.149776 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:28.152498 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:28.158504 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:28.164032 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:28.186292 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:28.226203 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:28.400199 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:28.408811 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:28.433758 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:28.436074 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:28.441066 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:28.496302 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:28.519930 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:28.874282 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:28.885107 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:28.898698 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:28.899205 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:28.930967 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:28.981775 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:29.090559 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:29.134299 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:29.140759 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:29.162662 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:29.165673 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:29.195537 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:29.339152 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.372064 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:29.377713 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:29.382635 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:29.394300 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:29.394474 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:29.395623 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:29.398329 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:29.421667 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:29.421828 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:29.444527 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:26:29.538272 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.619302 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:29.656448 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:29.663555 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.689512 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:26:29.703373 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:29.703928 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:29.713891 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:29.720937 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:29.759796 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.779023 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:29.805816 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:29.806883 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.892931 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.947885 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:29.978062 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:30.062067 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:30.176172 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:30.177115 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:30.190847 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:30.191164 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:30.194876 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.205382 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:30.295120 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:30.417124 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:30.425884 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:30.429384 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.431860 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:30.449162 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:30.455452 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:30.565821 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:30.658706 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.678599 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.680130 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:30.691346 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:30.703851 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:30.704154 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:30.745326 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:30.917081 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.921087 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:30.934612 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:30.936344 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:30.944272 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:30.978077 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:31.016199 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:26:31.407415 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:31.418088 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:31.420797 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:31.424410 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:31.434391 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:31.439162 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:31.472254 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:26:31.496232 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:31.531935 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:26:31.659111 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:31.667232 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:31.674675 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:31.674721 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:31.675376 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:31.836865 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:31.867585 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:31.949323 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:31.949693 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:26:31.965890 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:31.966327 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:31.979597 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:31.987776 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:31.993465 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:32.039651 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:32.076893 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.092597 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:32.117187 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:32.202077 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.228016 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:32.231462 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:32.250692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:32.251495 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:32.252414 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:32.259070 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:32.411486 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.423358 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.432653 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:32.444378 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:26:32.453851 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:32.465456 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.555781 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:32.670727 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:32.716842 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:32.726295 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:32.732730 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:32.734556 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:32.740322 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:32.837484 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:32.907505 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:26:32.908739 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:32.990622 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:32.990859 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:32.996672 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:32.999483 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:33.013370 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:33.014364 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:33.101830 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:33.189239 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:33.227322 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:33.236147 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:33.236613 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:33.244845 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:33.248032 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:33.410846 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:33.483788 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:33.498175 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:33.506470 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:33.519387 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:33.534096 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:33.779183 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:33.812029 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:33.973977 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:33.987720 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:34.001421 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:34.011158 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:34.023593 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:34.211514 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:34.249096 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:34.249989 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:34.271180 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:34.272272 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:34.329038 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:34.358989 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:34.364274 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:26:34.497887 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:34.507489 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:34.510797 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:34.518225 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:34.518741 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:34.551285 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:34.595144 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:34.644758 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:34.670519 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:34.671730 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:34.681569 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:34.682596 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:34.721518 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:34.728569 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:34.742064 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:34.745170 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:34.762514 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:34.765450 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:26:34.768249 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:34.830689 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:26:34.876761 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:34.886962 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:34.903301 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:26:35.040777 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:35.164793 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:35.200918 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:26:35.252072 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:35.257797 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:35.264045 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:35.272972 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:35.276938 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:35.304383 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:35.407986 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:26:35.494568 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:35.498947 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:35.498947 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:35.517681 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:35.539696 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:35.553463 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:26:35.564689 11632 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:35.565050 11632 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:35.565914 11632 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:35.566959 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:35.568563 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:35.568706 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:35.568986 11632 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:26:35.668538 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:35.688700 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:35.719017 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:35.739130 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:35.740912 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:35.743461 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:35.772122 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:35.830662 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:36.073311 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:36.078478 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:36.080883 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:36.088160 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:36.094646 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:36.097558 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:26:36.153118 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:36.220203 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:36.557891 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:36.577716 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:36.577965 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:36.598788 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:36.611958 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:36.628229 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:26:36.704682 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:36.756899 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:36.771819 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:26:36.829833 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:36.843444 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:36.844914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:36.863085 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:36.875273 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:36.948613 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:36.967947 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:37.044939 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:37.054320 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:37.066614 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:37.079481 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:37.088064 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:37.090978 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:37.094388 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:37.128845 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:37.149237 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:37.156246 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:37.156977 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:26:37.165659 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:37.165836 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:37.309083 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:37.312378 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:37.316254 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:37.334836 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:26:37.336638 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:37.343407 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:37.355288 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:37.376753 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:37.461112 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:37.722326 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:37.766860 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:37.768600 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:37.776384 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:37.799737 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:37.834806 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:37.911094 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:38.024947 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:38.046123 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:38.068549 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:38.076433 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:38.084080 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:38.084583 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:38.102517 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:38.131628 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:38.195474 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:26:38.302547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:38.308295 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:38.331950 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:38.336163 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:38.339910 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:38.344259 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:38.364228 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:38.410077 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:38.549942 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:38.555581 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:38.564929 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:38.578473 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:38.588233 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:38.594753 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:38.723352 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:39.029258 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.050967 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:39.053787 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:39.056545 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.084542 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:39.124702 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:39.214124 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:39.263983 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:39.266541 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:39.290426 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.305691 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:39.328830 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:39.344146 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.367543 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:39.371501 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:39.445907 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:39.476157 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:39.483731 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:26:39.560926 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:39.561383 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:39.566047 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:39.572664 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.573866 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:39.583164 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:39.603153 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:39.604116 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:39.659307 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:39.679705 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:39.783823 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:39.793681 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:39.813423 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:39.814936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:39.842648 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:39.875303 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:39.900938 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:39.982980 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:40.082993 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:40.140655 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:26:40.141857 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:40.185145 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:40.269191 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:40.320983 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:40.346694 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:40.349973 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:40.364169 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:40.370765 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:40.384253 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:40.582064 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:40.607077 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:40.615182 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:40.625628 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:40.640594 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:40.641321 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:40.656913 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:40.667230 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:40.801268 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:40.880422 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:40.881036 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:40.887595 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:40.897433 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:40.926589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:41.111567 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 08:26:41.122100 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:41.131567 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:41.146087 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dc49c050280346db88d1323e30513c80: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:26:41.155220 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:41.166909 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:41.168697 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:41.186077 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:41.210557 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:41.307677 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:41.419354 14157 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:41.419572 14157 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:41.420387 14157 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:41.421463 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:41.421972 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:41.422075 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:41.422340 14157 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:26:41.597719 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:41.597800 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:41.616039 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:41.616631 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:41.651647 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:41.672755 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:41.693049 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:26:41.697490 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:41.731526 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:41.831421 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:26:41.831600 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:41.867691 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:41.881953 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:41.899708 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:41.901682 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:41.909533 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:41.911882 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:42.003317 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.069974 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:42.084992 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.112457 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:26:42.113991 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:42.115016 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:42.122453 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:42.124951 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:42.133733 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:42.183555 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:42.204807 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.253418 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:42.328824 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:42.378623 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:42.393224 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:42.394110 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:42.396620 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:42.418603 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:42.421622 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:42.454351 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.555087 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.719965 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:42.806743 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:42.867599 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:42.869055 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:42.897178 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:42.914489 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:42.923576 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:42.957743 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:43.131901 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:43.146682 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:43.150993 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:43.162087 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:43.167918 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:43.171952 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:43.298717 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:43.368290 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:43.374436 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:43.378753 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:43.387231 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:26:43.393761 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:43.399678 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:43.410172 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:43.425902 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:43.632980 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:43.634526 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:43.641661 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:43.653409 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:43.666522 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:43.812234 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:43.854187 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:43.906420 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:43.994518 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:26:44.073417 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:44.079802 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.082103 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:44.096266 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:44.113754 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:44.164016 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.303304 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:44.365679 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:44.401551 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:44.404904 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:44.405871 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.419477 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:44.423525 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:44.428292 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.493465 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:44.527815 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:44.528546 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:44.530946 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:44.636366 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:44.636701 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:44.636847 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:44.647010 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:44.656638 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.663131 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:44.682497 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:44.696473 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:44.723330 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:26:44.800670 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:44.898509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:44.900542 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:44.917295 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:44.920607 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:44.951385 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:44.951519 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:44.975028 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:45.108124 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:26:45.241401 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:45.403899 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.409070 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:45.432324 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:45.433401 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.443171 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:45.513065 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:45.549616 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:45.560094 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:45.655228 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.666029 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:45.669673 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:45.691744 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.698393 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:45.777748 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:45.784972 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:45.927299 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.934741 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:45.956357 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:45.963837 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:45.976179 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:46.081351 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:46.146270 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:26:46.176548 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:26:46.193790 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:46.199699 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:46.221369 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:46.221541 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:46.222227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:46.321110 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:46.511354 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:46.537739 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:46.670974 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:46.672703 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:46.684813 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:46.684979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:46.707365 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:46.867569 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:46.876360 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:46.912560 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:46.970157 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:46.976639 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:46.981580 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:46.982985 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:46.984920 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:47.014612 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:26:47.015703 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:47.026473 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:47.054849 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:26:47.095693 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:26:47.104446 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:47.129452 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:47.134974 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:47.163211 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:47.167964 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:47.174275 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:26:47.202399 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:47.210172 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:47.215132 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:47.219851 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:47.233613 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:47.282074 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:47.445008 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:47.445065 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:47.450974 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:47.451365 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:47.468418 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:47.486070 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:47.507308 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:47.600441 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:47.899773 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:47.915921 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:47.917579 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:47.922245 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:47.929751 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:47.949240 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:47.953475 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:48.085307 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:48.093304 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:48.125170 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:48.142181 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:48.153645 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:48.163944 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:48.175969 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:48.204838 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:48.384549 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:48.395738 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:48.396165 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:48.397423 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:48.398557 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:48.418761 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:48.471741 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:26:48.645002 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:48.649822 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:48.661826 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:48.666074 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:48.683429 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:48.696372 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:48.865732 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:48.945248 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:48.995501 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:49.106799 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:49.109272 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:49.109930 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.119995 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.124220 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:49.148524 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:49.323450 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:49.363318 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:49.369215 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:49.372280 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:49.378351 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.382355 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:49.397073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:49.415903 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:49.468436 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:49.485323 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:49.609942 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:49.628007 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.630251 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:49.636179 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:49.636993 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.642104 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:49.685119 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:49.738605 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:49.797606 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:49.854182 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:49.870568 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:26:49.879302 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:49.897023 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:49.904311 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:49.912250 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:49.915478 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:49.931851 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:49.938100 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:49.951519 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:50.028187 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:50.124517 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:26:50.346940 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:50.348832 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:50.379065 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:50.379894 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:50.381016 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:50.402418 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:50.409056 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:50.516335 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:50.592414 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:50.594470 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:50.613677 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:50.631090 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:50.635272 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:50.649259 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:50.757901 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:50.805600 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:50.886126 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:50.909318 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:50.912030 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:50.913874 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:50.915422 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:51.047581 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:26:51.141613 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:51.161545 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:51.161846 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:51.161965 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:51.173807 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:51.302333 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:26:51.361847 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:51.406777 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:51.457957 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:26:51.581699 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:51.593290 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:51.596174 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:51.602001 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:51.622972 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:51.624279 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:51.813155 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:51.825225 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:51.863543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:51.883001 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:51.885921 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:51.908370 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:51.911618 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:51.917764 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:52.081413 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:26:52.082604 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:52.108004 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:52.108028 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:52.151289 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:52.154747 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:52.158150 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:52.163210 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:52.178004 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:52.229362 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:52.269584 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:52.277748 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:52.287943 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:52.331645 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:26:52.333014 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:52.377089 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:52.381181 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:52.402920 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:52.407816 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:52.421887 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:52.448944 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:52.449741 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:52.455685 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:52.563433 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:26:52.854645 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:52.862493 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:52.874060 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:52.898111 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:52.918450 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:52.950400 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:52.958875 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:53.006192 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:53.037623 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:53.137755 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:53.142002 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:53.154076 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:53.170645 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:53.192487 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:53.302796 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:53.341521 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:53.399453 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:53.402801 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:53.418612 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:53.421298 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:53.421484 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:53.425716 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:53.655324 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:53.658833 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:53.662992 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:53.686475 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:53.687430 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:53.803589 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:53.811074 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:26:53.848209 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:54.002318 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:54.048833 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:54.106941 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:54.110572 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:54.126956 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:54.129469 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:54.129895 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:54.175540 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:54.264989 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:54.346681 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:54.373955 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:54.391520 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:54.393909 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:54.399828 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:54.407706 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:54.466121 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:54.545086 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:54.584957 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:54.592713 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:54.598415 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:54.607735 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:54.618460 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:54.639178 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:54.656244 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:54.671358 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:54.748175 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:54.780892 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:54.806370 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:54.855019 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:54.885987 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:54.916164 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:54.924459 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:54.925607 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:54.931622 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:54.948361 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:55.012367 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:55.058666 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:26:55.146906 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:26:55.201663 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:55.342119 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:55.363888 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:55.375597 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:55.395524 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:55.408416 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:55.409926 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:55.435914 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:55.608317 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:55.615314 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:55.615666 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:55.644589 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:55.644605 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:55.660399 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:55.660722 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:55.857102 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:55.871120 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:55.883649 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:55.899479 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:55.905007 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:55.931190 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:55.947593 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:56.143812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:56.153673 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:56.161257 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:56.167807 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:56.175338 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:56.305353 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:26:56.307416 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:56.372593 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:56.513515 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:26:56.572682 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:56.573179 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:56.598067 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:56.598371 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:56.598891 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:56.715548 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:56.797586 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:56.842235 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:56.878225 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:56.881613 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:56.887468 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:56.912423 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:56.940722 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:56.969194 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:57.005102 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:57.020220 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:57.124126 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:57.140089 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:57.148419 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:57.151207 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:57.152417 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:57.217137 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:57.253296 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:26:57.257047 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:57.297144 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:57.328387 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:57.335213 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:26:57.403584 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:57.416378 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:57.421195 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:57.421919 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:57.423825 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:57.455713 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:57.485725 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:57.498185 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:57.500725 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:57.536481 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:26:57.537864 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:57.800884 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:57.840075 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:26:57.889376 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:57.912896 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:57.924237 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:26:57.925792 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:57.929231 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:57.976051 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:58.064572 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:58.173590 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:58.173590 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:58.188683 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:26:58.191295 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:58.194761 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:58.196267 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:58.414002 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:58.436232 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:58.441124 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:58.442381 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:58.464756 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:58.601444 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:26:58.655901 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:58.662523 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:58.665020 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:58.674466 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:58.675516 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:58.694406 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:58.771373 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:59.055883 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:26:59.128541 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.135839 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:59.138667 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:59.145419 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:59.148705 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:59.166684 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.188848 15916 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:26:59.189056 15916 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:26:59.189955 15916 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:26:59.191162 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:26:59.192044 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:59.192179 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:26:59.193866 15916 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:26:59.194761 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:59.204926 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:26:59.244092 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:26:59.357110 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:26:59.403352 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.408483 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:59.408476 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:59.415489 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:59.439569 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:59.518419 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:26:59.547816 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:59.608733 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:26:59.629976 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.636669 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:26:59.641782 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:26:59.653618 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:59.657552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.665331 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:26:59.716303 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:26:59.726891 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:26:59.733528 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:26:59.907913 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:26:59.911329 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:26:59.924192 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:26:59.945762 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:26:59.946602 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:26:59.964576 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:26:59.992403 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:00.015677 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:27:00.019649 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:27:00.055948 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:00.073292 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:27:00.135962 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:00.186496 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:00.299057 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:27:00.382838 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:00.383330 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:00.390138 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:00.405720 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:00.410818 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:00.500557 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:00.587036 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:00.610237 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:00.624007 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:00.631975 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:00.638332 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:00.654100 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:00.825721 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:27:00.849699 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:00.868703 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:00.869329 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:00.876647 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:00.916162 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:00.940480 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:27:01.126705 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:01.143951 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:01.145449 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:01.146626 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:01.172073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:01.198268 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:01.252828 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:01.345824 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:27:01.390254 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:01.412761 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:01.530369 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:27:01.578125 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:01.596051 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:01.600894 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:01.605373 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:01.624163 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:01.651787 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:01.843876 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:01.858302 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:01.861963 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:27:01.869478 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:01.874879 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:01.887758 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:01.932770 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:01.999733 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:27:02.086453 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:02.096037 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:27:02.115392 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:02.131505 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:02.139039 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:02.144838 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:02.159545 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:02.182718 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:02.300712 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:02.311162 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:02.346802 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:02.357556 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:27:02.359288 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:02.377148 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:27:02.385767 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:02.398010 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:02.403007 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:02.409034 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:02.431525 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:27:02.438148 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:02.452726 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:02.618670 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:02.788038 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:02.886427 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:02.895503 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:02.902384 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:27:02.910943 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:02.944098 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:02.959239 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:02.987349 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:03.055146 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:03.157636 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:03.192845 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:03.201635 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:03.212641 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:03.226179 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:03.227052 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:03.234128 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:03.346465 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:27:03.494290 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:03.499047 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:03.507985 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:03.510092 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:03.531677 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:03.562674 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:03.767410 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:03.787479 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:03.794190 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:03.807288 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:03.835711 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:03.844801 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.013685 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:04.108219 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:04.127714 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:04.186488 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:04.236181 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:27:04.254587 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:04.285241 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:04.307701 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:04.313881 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:04.325866 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.339411 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:04.340157 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.545637 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:04.547217 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:04.561069 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:04.562539 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:04.584719 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:04.588121 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.660363 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:04.669701 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:27:04.805850 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.814057 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:04.818140 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:04.843736 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:04.854151 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:04.870486 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:04.897146 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:04.903447 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:04.924824 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:27:04.931281 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:05.045977 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:05.046727 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:05.064643 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:05.096204 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:05.133111 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:05.134660 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:05.143512 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:05.165227 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:05.189960 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:05.195981 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:05.244304 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:27:05.388394 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:05.448809 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:05.587659 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:05.593768 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:05.608767 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:05.617990 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:05.635811 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:05.636747 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:05.668617 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:05.876410 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:05.883069 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:05.898409 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:05.924011 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:05.934715 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:05.941124 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:06.011855 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:06.140622 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:06.156939 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:06.165938 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:06.181445 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:06.188844 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:06.253888 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:06.360958 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:27:06.392132 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:06.396304 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:06.396852 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:06.403787 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:06.415789 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:06.441489 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:06.590286 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:06.615581 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:06.693149 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:06.831868 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:06.882863 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:06.883383 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:06.884387 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:06.886200 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:06.891872 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:06.925693 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:06.929848 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:07.135447 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:07.136252 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:07.147210 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:07.148731 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:07.168052 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:07.176752 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:27:07.177148 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:07.330924 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:27:07.334466 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:07.385193 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:07.391265 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:07.412889 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:07.427735 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 45931.7us
I20240502 08:27:07.431620 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:07.440800 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:07.515748 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:07.515839 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:07.532137 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:27:07.566749 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:07.569038 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:07.670117 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:07.672061 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:07.685122 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:07.686143 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:07.689498 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:07.702041 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:07.716614 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:07.748021 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:07.810180 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:07.874269 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:07.936208 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:07.979091 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:08.125480 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:08.126376 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:08.126451 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:08.140542 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:08.143869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:08.200815 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:08.241732 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:27:08.373008 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:08.380297 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:08.388720 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:08.396981 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:08.414137 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:08.486196 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:08.589671 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:08.608762 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:08.613790 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:08.618755 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:08.632875 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:08.635757 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:08.639847 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:08.881990 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:08.892976 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:08.908960 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:08.910391 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:08.920447 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:09.099469 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:09.177559 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:09.230564 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:09.341203 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:09.359917 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:09.361837 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:09.376844 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:09.377064 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:09.386371 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:09.472575 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:09.484448 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:09.530054 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:09.565182 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:09.598217 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:09.611265 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:09.641996 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:09.647368 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:09.655928 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:09.664906 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:09.689085 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:09.879707 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:09.887209 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:09.904383 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:27:09.904824 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:09.907449 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:09.920404 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:09.924674 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:27:09.936996 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:09.943645 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:10.069236 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:10.079010 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:10.093875 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:10.129352 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.144413 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:10.146385 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.150377 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:10.167956 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:10.246088 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:10.293958 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:27:10.295014 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:10.341298 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:10.480608 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:10.582614 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.593345 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:10.614344 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:10.623554 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.628082 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:10.646996 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:10.650831 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:10.712121 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:10.837983 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.872426 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:10.876943 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:10.904561 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:10.926484 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:11.002933 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:11.050911 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:11.114717 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:11.122313 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:11.137390 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:11.137856 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:11.144938 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:11.149533 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:11.372893 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:11.385702 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:11.407035 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:11.407366 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:11.410300 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:11.540587 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:27:11.541497 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:11.547924 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:11.581178 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:11.644369 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:11.737974 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:11.841482 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:11.844000 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:11.866602 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:11.888494 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:11.889335 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:12.034564 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:12.090796 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:12.109375 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:12.140846 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:12.155429 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:12.157824 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:12.161334 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:12.183969 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:27:12.185962 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:12.207607 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:12.343997 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:12.357043 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:12.392660 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:12.400244 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:12.401783 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:12.415489 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:12.427342 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:12.480197 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:27:12.496461 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:12.565081 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:27:12.601444 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:12.614358 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:12.621358 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:12.625162 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:12.628161 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:12.633716 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:12.641498 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:12.756554 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:12.796921 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:12.826251 15916 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:27:12.826460 15916 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:27:12.827179 15916 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:27:12.828253 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:27:12.828716 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:12.828809 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:27:12.829022 15916 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:27:12.860498 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:12.954273 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:27:12.955490 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:27:13.003362 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:13.067994 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:13.074201 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:13.085601 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:13.101830 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:13.104841 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:13.124577 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:13.225647 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:13.246048 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:13.258030 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:13.311951 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:13.339181 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:13.365196 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:13.392027 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:13.463526 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:13.476269 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:13.632962 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:13.635449 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:13.636555 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:13.637566 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:13.649438 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:13.680871 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:13.883982 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:13.890056 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:13.894570 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:13.916513 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:13.924126 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:13.936815 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:13.993801 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:14.002030 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:27:14.052809 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:27:14.084137 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:14.216205 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:14.319070 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:14.340637 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:14.346825 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:14.397898 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:14.399020 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:14.411058 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:14.417899 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:14.434041 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 47376.2us
I20240502 08:27:14.442140 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:14.613914 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:14.643355 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:14.661103 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:14.662115 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:14.678311 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:14.690454 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:14.696561 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:14.700990 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:14.737485 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:14.772833 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:14.778204 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:14.800186 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:14.901224 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:14.960829 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:14.997673 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:27:14.999030 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:15.134801 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:15.136442 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:27:15.176024 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:15.208694 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:15.210997 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:15.215736 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:15.227967 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:15.245906 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:15.251960 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:15.269914 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:15.327611 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:15.481709 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:27:15.517686 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:15.546317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:15.547541 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:15.550983 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:15.569489 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:15.572270 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:15.578001 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:15.593164 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:15.659601 23922 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskivhcCF/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 08:20:17.427881 23925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:20:17.428491 23925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:20:17.464079 23925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 08:20:20.639816 24350 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 48847.6us
W20240502 08:20:20.726680 24350 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 69693.7us
W20240502 08:20:20.814886 24350 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 43681.5us
I20240502 08:20:21.285135 23925 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:20:22.860527 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.860879 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.861017 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.861218 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.861310 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:22.861413 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.861506 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.861653 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.861785 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.861907 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.862035 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:22.862146 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:22.862238 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:22.862339 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:20:23.087188 23925 rebalancer_tool.cc:1084] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:20:23.087519 23925 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:25.140630 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.158695 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.163806 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.170989 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.171236 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.171433 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.171586 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.171744 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.171926 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.172119 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:25.172259 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:25.172408 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:25.172520 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:25.172645 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:25.236218 23925 rebalancer_tool.cc:1074] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:25.641441 23925 rebalancer_tool.cc:1084] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:20:25.641604 23925 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:27.657778 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:27.658026 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:27.658166 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d0ce58fc888404ab140e25ec9c92a9b: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:27.658241 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:27.658396 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:27.658533 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:27.658686 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:27.658787 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:27.658883 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:27.658974 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d3160598c3b46be94de175cb8b05c0a: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 95f32202b9db457fb09872f6f51ea8ba
W20240502 08:20:27.659082 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0c5d895e104248b0a1ccfad7d1d590de: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:27.659179 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:27.659377 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:27.659475 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7fcdc1cbce1641c3b983eb474059b2c5: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server fd5e97ae7e344e4687b15ca0cde8d0ba
W20240502 08:20:27.659582 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
I20240502 08:20:28.189291 23925 rebalancer_tool.cc:1074] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:28.811151 23925 rebalancer_tool.cc:1074] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
W20240502 08:20:29.767556 25795 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.142s sys 0.299s
W20240502 08:20:29.767946 25795 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.580s user 0.142s sys 0.299s
I20240502 08:20:30.092175 23925 rebalancer_tool.cc:1074] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:30.476133 23925 rebalancer_tool.cc:1074] tablet b88c4a1a36424086aad89b7585c17960: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:20:30.527861 23925 rebalancer_tool.cc:1074] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:31.204224 23925 rebalancer_tool.cc:1074] tablet e0241228bd204ca2a8ad2c57b6eeb543: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:31.982009 23925 rebalancer_tool.cc:1074] tablet 59f5d97e6e0a4755be41699246907e55: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:20:32.475168 23925 rebalancer_tool.cc:1074] tablet dc49c050280346db88d1323e30513c80: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:20:32.643271 23925 rebalancer_tool.cc:1074] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:32.960709 23925 rebalancer_tool.cc:1074] tablet a20b2fd9f1ce488ba7793ed7ac268c16: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:33.069494 23925 rebalancer_tool.cc:1084] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> '20b637fbec5444dd91b3af9cdda482f5' move ignored: Illegal state: tablet 80b3ba1d834a4314b63f336675845441: replica 20b637fbec5444dd91b3af9cdda482f5 is already present
I20240502 08:20:33.069664 23925 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:20:33.787678 26504 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 117412us
W20240502 08:20:33.904147 26504 meta_cache.cc:1097] Time spent processing table locations response: real 0.063s user 0.000s sys 0.000s
W20240502 08:20:33.906224 26504 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 65126.8us
W20240502 08:20:34.127965 26504 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 63079.1us
W20240502 08:20:34.175093 26504 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:44613, user_credentials={real_user=slave}} blocked reactor thread for 46616.7us
W20240502 08:20:35.035984 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.036198 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server 13e764b27ed449c09aee719cdb2597f8 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.036365 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 32bbe3ff8a894fef891c38475cbd42d8: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:35.036530 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f04d6b2034c9494daf0dd7d90ed835a6: could not find any suitable replica to move from server fd5e97ae7e344e4687b15ca0cde8d0ba to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.036620 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7dfe697bbc4946eebea2280bb24e7506: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server 4b9bfd2541144dad86507451343640d3
W20240502 08:20:35.036834 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ec7f747ed54e4636a94dd2cd32110cfb: could not find any suitable replica to move from server cfff665685a14d8796aece26addab59c to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:35.036978 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b6feb10e86734c6988fe85ecc1269f2d: could not find any suitable replica to move from server 0f655ab8f20d4ba482c3173f9594b98b to server 20b637fbec5444dd91b3af9cdda482f5
W20240502 08:20:35.037113 23925 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe5182679a84496691f023e5b676fd50: could not find any suitable replica to move from server 60677121510e47a5b51d1eb22f9aa238 to server fd5e97ae7e344e4687b15ca0cde8d0ba
I20240502 08:20:35.098625 23925 rebalancer_tool.cc:1074] tablet 9c0a886b3ba04737a6988c3b58c9698e: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:20:35.133311 23925 rebalancer_tool.cc:1074] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:35.391188 23925 rebalancer_tool.cc:1074] tablet d4d933eacd944d84b39f0cb9c45641f8: '13e764b27ed449c09aee719cdb2597f8' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:20:47.884325 23925 rebalancer_tool.cc:1136] tablet 3898bfdb9c3d4a05af78f79fd2de61e2: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:48.111871 23925 rebalancer_tool.cc:1136] tablet 2de41d6c6de0400397645c05a0fa17de: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:50.671862 23925 rebalancer_tool.cc:1136] tablet 6f04a3972e384fa3b0ffe0cb8a4f35aa: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:20:50.731504 23925 rebalancer_tool.cc:1136] tablet 9c0a886b3ba04737a6988c3b58c9698e: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:20:50.894855 23925 rebalancer_tool.cc:1136] tablet efec2cc693bd4dd992d0bec22fca5419: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:50.972157 23925 rebalancer_tool.cc:1136] tablet b88c4a1a36424086aad89b7585c17960: '13e764b27ed449c09aee719cdb2597f8' -> '4b9bfd2541144dad86507451343640d3' move completed: OK
I20240502 08:20:52.883646 23925 rebalancer_tool.cc:1136] tablet 1648586b889b4a1e9ac34481cec6257a: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:20:53.083581 23925 rebalancer_tool.cc:1074] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:53.251756 23925 rebalancer_tool.cc:1074] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:53.468377 23925 rebalancer_tool.cc:1074] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:20:53.657691 23925 rebalancer_tool.cc:1074] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:53.752367 23925 rebalancer_tool.cc:1136] tablet 07f5c2dfcd974ee2b8720e9eb052c3f7: '60677121510e47a5b51d1eb22f9aa238' -> '' move completed: OK
I20240502 08:20:53.804101 23925 rebalancer_tool.cc:1136] tablet 8a45b1da348e4bb89f021df103a0603f: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:20:53.894156 23925 rebalancer_tool.cc:1136] tablet a20b2fd9f1ce488ba7793ed7ac268c16: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:54.095503 23925 rebalancer_tool.cc:1136] tablet e0241228bd204ca2a8ad2c57b6eeb543: '60677121510e47a5b51d1eb22f9aa238' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:54.178972 23925 rebalancer_tool.cc:1136] tablet 76bb1e3e2b9d4e31bf93ac1ce08a7833: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:20:54.608490 23925 rebalancer_tool.cc:1136] tablet 7b3f6a0b2561419ca6bd832fd65e3196: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:20:54.766228 23925 rebalancer_tool.cc:1136] tablet dc49c050280346db88d1323e30513c80: '0f655ab8f20d4ba482c3173f9594b98b' -> '95f32202b9db457fb09872f6f51ea8ba' move completed: OK
I20240502 08:20:54.844192 23925 rebalancer_tool.cc:1136] tablet 59f5d97e6e0a4755be41699246907e55: '0f655ab8f20d4ba482c3173f9594b98b' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:20:54.956961 23925 rebalancer_tool.cc:1136] tablet d8042f6ab6404c5a9f8cce0b8f971542: '60677121510e47a5b51d1eb22f9aa238' -> '20b637fbec5444dd91b3af9cdda482f5' move completed: OK
I20240502 08:20:55.333544 23925 rebalancer_tool.cc:1136] tablet c4862ea2f67c43a28689b464d8ec96e6: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:20:55.548249 23925 rebalancer_tool.cc:1136] tablet d4d933eacd944d84b39f0cb9c45641f8: '13e764b27ed449c09aee719cdb2597f8' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move completed: OK
I20240502 08:20:55.634423 23925 rebalancer_tool.cc:1074] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move scheduled
I20240502 08:20:55.769397 23925 rebalancer_tool.cc:1074] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:55.923177 23925 rebalancer_tool.cc:1074] tablet cec56facd7f740aeb42bc7c32495dae3: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:55.965302 23925 rebalancer_tool.cc:1074] tablet 203e7914da3f4494b239c6603513caee: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move scheduled
I20240502 08:20:56.126461 23925 rebalancer_tool.cc:1074] tablet 12c17796e5d04014aafdac46984f4ba9: '13e764b27ed449c09aee719cdb2597f8' -> '' move scheduled
I20240502 08:20:56.280411 23925 rebalancer_tool.cc:1136] tablet 12c17796e5d04014aafdac46984f4ba9: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:00.759872 23925 rebalancer_tool.cc:1136] tablet 182a6f2df91242cfa10f05ef788a19cb: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:00.962462 23925 rebalancer_tool.cc:1136] tablet 599d5781f1e24ae9ad425a34d88c6b68: 'cfff665685a14d8796aece26addab59c' -> '' move completed: OK
I20240502 08:21:01.035957 23925 rebalancer_tool.cc:1136] tablet 203e7914da3f4494b239c6603513caee: '0f655ab8f20d4ba482c3173f9594b98b' -> '' move completed: OK
I20240502 08:21:01.116005 23925 rebalancer_tool.cc:1136] tablet cec56facd7f740aeb42bc7c32495dae3: '13e764b27ed449c09aee719cdb2597f8' -> '' move completed: OK
I20240502 08:21:02.756047 23925 rebalancer_tool.cc:1074] tablet 80b3ba1d834a4314b63f336675845441: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:03.048650 23925 rebalancer_tool.cc:1074] tablet dc49c050280346db88d1323e30513c80: '95f32202b9db457fb09872f6f51ea8ba' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:21:03.629645 23925 rebalancer_tool.cc:1074] tablet b88c4a1a36424086aad89b7585c17960: '0f655ab8f20d4ba482c3173f9594b98b' -> '13e764b27ed449c09aee719cdb2597f8' move scheduled
I20240502 08:21:04.411751 23925 rebalancer_tool.cc:1074] tablet d96a7174eb5442f2b0cc0cafc3cc4c80: 'cfff665685a14d8796aece26addab59c' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:21:04.899835 23925 rebalancer_tool.cc:1074] tablet a20b2fd9f1ce488ba7793ed7ac268c16: 'fd5e97ae7e344e4687b15ca0cde8d0ba' -> '4b9bfd2541144dad86507451343640d3' move scheduled
I20240502 08:21:04.988030 23925 rebalancer_tool.cc:1074] tablet 599d5781f1e24ae9ad425a34d88c6b68: '60677121510e47a5b51d1eb22f9aa238' -> '' move scheduled
I20240502 08:21:05.661831 23925 rebalancer_tool.cc:1074] tablet ebf79c69fd614284ba45fd052cac7ebf: '95f32202b9db457fb09872f6f51ea8ba' -> '20b637fbec5444dd91b3af9cdda482f5' move scheduled
I20240502 08:21:05.976517 23925 rebalancer_tool.cc:1074] tablet 9c0a886b3ba04737a6988c3b58c9698e: '4b9bfd2541144dad86507451343640d3' -> '0f655ab8f20d4ba482c3173f9594b98b' move scheduled
I20240502 08:21:06.277422 23925 rebalancer_tool.cc:1074] tablet 2bfc3eaed583458b8381d4e7b4d36aef: 'cfff665685a14d8796aece26addab59c' -> 'fd5e97ae7e344e4687b15ca0cde8d0ba' move scheduled
I20240502 08:21:06.346046 23925 rebalancer_tool.cc:1074] tablet d8042f6ab6404c5a9f8cce0b8f971542: '20b637fbec5444dd91b3af9cdda482f5' -> '' move scheduled
I20240502 08:21:06.669618 23925 rebalancer_tool.cc:1074] tablet 182a6f2df91242cfa10f05ef788a19cb: '4b9bfd2541144dad86507451343640d3' -> '95f32202b9db457fb09872f6f51ea8ba' move scheduled
I20240502 08:21:07.384155 23925 rebalancer_tool.cc:1074] tablet 203e7914da3f4494b239c6603513caee: 'cfff665685a14d8796aece26addab59c' -> '4b9bfd2541144dad86507451343640d3' move scheduled
W20240502 08:21:08.824501 32035 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:21:08.573834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:41299 (local address 127.0.0.1:34046)
0502 08:21:08.575060 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0502 08:21:08.575080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 08:21:08.575215 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:21:08.822974 (+247759us) negotiation.cc:327] Negotiation complete: Network error: Clien
W20240502 08:27:15.798774 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:15.832278 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:15.989814 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:16.048493 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:16.050195 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.052975 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:16.053717 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.054352 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:16.225198 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:16.252846 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:16.288491 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.311465 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:16.311848 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:16.327908 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.354243 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:16.358960 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:16.381887 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:16.469884 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:27:16.553186 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.558601 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:16.571908 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:16.584072 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:16.585922 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:16.596799 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:27:16.712201 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:16.825333 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.832836 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:16.843183 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:16.892237 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:16.893893 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:16.896975 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:16.979337 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:17.154127 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:17.166805 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:17.179819 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:17.334784 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:17.340811 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:17.351457 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:17.367630 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:17.369832 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:17.383857 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:17.474651 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:27:17.529821 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:17.583238 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:17.595814 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:17.602880 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:17.607784 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:17.626726 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:17.627601 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:17.632591 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:17.655298 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:17.662006 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:17.751300 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:17.856370 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:17.862076 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:17.874151 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:17.875531 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:17.876969 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:17.895766 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:18.041841 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:18.057333 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:18.121117 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:18.143154 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:18.154974 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:18.156080 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:18.168962 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:18.180783 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:18.209986 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:18.233664 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:18.285312 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:18.459864 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:18.585162 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:18.588760 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:18.597452 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:18.609117 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:18.641438 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:18.643146 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:18.692332 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:18.810870 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:18.874414 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:18.877732 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:18.883455 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:18.889796 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:18.895427 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:18.914932 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:18.929610 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:19.139070 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:19.155936 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:19.165068 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:19.176203 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:19.195103 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:19.209872 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:19.253473 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:19.272282 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:19.412396 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:19.425016 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:19.434993 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:19.439602 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:19.454182 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:19.518527 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:27:19.560138 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:19.588035 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:19.629509 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:19.877698 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:19.879758 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:19.884241 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:19.884917 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:19.895375 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:27:19.900300 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:19.930775 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:19.937592 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:19.959282 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:20.104964 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:20.105890 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:20.120416 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.137169 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:20.140622 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:20.145730 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.174975 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:20.201905 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:20.248749 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:20.284206 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:20.362732 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:20.397090 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.421474 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:20.422333 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.429426 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:20.446725 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:20.486089 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:20.557796 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:20.640619 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.651129 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:20.672430 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:20.675858 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:20.687057 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:20.751982 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:20.769011 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:20.865769 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:20.904718 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:20.958029 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:20.976646 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:21.094168 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:21.107010 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:21.108610 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:21.115640 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:21.150667 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:21.173012 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:21.173254 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:21.354524 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:21.355976 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:21.368324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:21.371714 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:21.379446 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:21.406940 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:21.439453 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:21.635671 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:21.647882 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 08:27:21.654537 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:21.663157 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:21.666605 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:21.700448 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:21.832435 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:21.881083 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:21.917160 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:21.917547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:21.932526 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:21.935462 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:21.939621 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:21.948083 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:21.979645 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:27:22.088358 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:22.178789 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:22.343952 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:27:22.356247 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:22.372016 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:22.392846 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:22.401360 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:22.416821 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:22.441006 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:22.475986 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:22.591910 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:22.648298 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:27:22.667966 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:22.678861 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:22.681977 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:22.683920 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:22.685400 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:22.716781 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:22.753159 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:22.804888 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:22.868721 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:22.907234 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:22.936278 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:22.939982 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:22.949831 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:22.954902 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:22.956843 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:22.956993 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:23.191119 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:23.208086 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:23.208472 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:23.210938 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:23.217654 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:23.233983 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:23.387033 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:23.387027 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:23.432693 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:23.486340 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:23.533996 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:23.626940 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:27:23.628531 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:23.628839 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:23.637831 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:23.646170 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:23.661816 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:23.878469 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:23.894599 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:23.897554 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:23.898623 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:23.927415 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:23.937677 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:23.952960 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:23.986043 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:23.992000 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 53073.2us
W20240502 08:27:23.997730 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:24.176990 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:24.177455 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:24.179163 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:24.197718 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:24.199535 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:24.361855 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:24.388727 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:24.411823 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:24.431435 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:24.433037 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:24.437541 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:24.455729 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:24.457283 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:24.672926 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:27:24.683929 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:24.889891 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:24.904629 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:24.922554 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:24.924218 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:24.936679 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:24.944924 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:27:24.947309 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:24.987818 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:25.125281 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:27:25.167101 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:25.178957 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:25.184515 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:25.195564 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:25.202515 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:25.301769 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.376175 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.395001 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:27:25.446532 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:25.447788 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:25.458134 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:25.459724 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:25.475591 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:25.480393 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:25.502924 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:27:25.507323 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:25.592595 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:25.593848 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.636562 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:25.700209 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:25.700430 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:25.723937 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:25.725353 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:25.725841 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:25.733474 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:25.831416 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.876222 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.887933 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:25.963500 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:26.125445 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:26.197885 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:27:26.215202 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:26.216573 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:26.219298 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:26.233780 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:26.259059 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:26.271786 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:26.355969 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:26.465306 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:26.471647 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:26.480623 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:26.492172 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:26.531862 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:26.617386 15916 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:27:26.617623 15916 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:27:26.618584 15916 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:27:26.619738 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:27:26.620075 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:26.620190 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:27:26.620518 15916 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:27:26.710117 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:26.723518 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:26.723657 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:26.730789 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:26.735599 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:26.748564 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:27:26.764021 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:26.831918 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:27:26.847883 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:26.911686 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:27.023928 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:27.030864 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:27.041370 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:27.042143 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:27.064690 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:27.142289 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:27.334335 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:27.443625 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:27:27.455217 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:27.469421 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:27.472664 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:27.472808 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:27.473659 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:27.474774 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:27.671334 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 08:27:27.679915 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:27.682492 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:27.697449 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:27.702013 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:27.708868 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:27.729568 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:27.736845 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:27.750947 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:27.805446 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:27.810066 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:27.903934 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:27.942415 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:27.954377 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:27.956023 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:27.977859 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:27.979135 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:28.019155 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:28.064502 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:28.143991 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:28.211071 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:28.213510 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:28.216218 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:28.220124 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:28.233034 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:28.241297 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:28.309942 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:28.428429 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:28.445060 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:28.639022 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:28.671619 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:28.675424 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:28.678213 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:28.685422 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:28.689062 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:28.711155 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:28.747473 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:28.951934 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:28.955060 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:28.976344 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:28.977890 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:28.978912 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:29.005864 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:29.063036 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:29.227381 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:29.229830 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:27:29.230547 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:29.233110 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:29.243553 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:29.252319 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:29.258589 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:29.286211 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:29.457680 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:29.468315 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:29.484357 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:29.491791 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:29.492096 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:29.493952 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:29.501793 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:29.557051 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:27:29.926345 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:29.933579 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:29.934620 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:29.935182 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:29.943334 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:29.947573 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:29.953755 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:27:29.971417 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:29.994021 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:30.093354 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:27:30.152272 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:30.162042 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:30.173090 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:30.197660 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:30.211594 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:30.240381 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:30.246698 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:30.385793 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:30.410147 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:30.410296 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:30.420982 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:30.427526 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:30.431915 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:30.434556 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:30.448854 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:30.523360 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 08:27:30.667732 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:30.688222 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:30.692471 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:30.692891 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:30.701248 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:30.723026 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:30.730247 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:30.745811 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:30.760200 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:30.841147 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:30.873128 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:30.991832 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:31.020110 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:31.127890 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:31.162144 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:31.170800 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:31.175550 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:31.185775 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:31.189641 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:31.233389 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:31.299917 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:27:31.429347 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:31.430049 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:31.439172 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:31.439924 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:31.440933 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:31.565248 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:31.621304 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:31.640421 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:31.647199 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:31.663295 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:31.663741 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:31.665388 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:31.667147 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:31.762320 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:31.764627 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:27:31.844739 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:27:31.902323 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:31.906394 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:31.907066 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:31.922745 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:31.923485 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:32.029079 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:32.118973 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:32.155385 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:32.353780 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:32.384243 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:32.387275 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:32.394758 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:32.416193 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:32.442690 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:32.450641 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:27:32.492408 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:32.530375 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:32.657418 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:32.663090 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:32.664324 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:32.665688 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:32.665709 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:32.681555 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:32.735073 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:32.816879 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240502 08:27:32.817914 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:32.934752 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:27:32.942579 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:32.947008 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:32.952472 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:32.958694 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:32.967906 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:32.968392 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:33.001246 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:33.084899 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:27:33.163159 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:33.168025 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:33.184377 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:33.192584 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:33.203754 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:33.302914 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:33.424611 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:33.493794 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:33.513789 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:33.530499 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:33.545250 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:33.606308 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:33.611579 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:33.632984 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:33.638173 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:33.653250 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:33.702081 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:33.813357 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:33.875258 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:33.897673 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:33.912506 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:33.935564 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:33.961650 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:34.095710 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:34.097900 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:34.107482 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:34.177495 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:34.183902 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:34.186555 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:34.204481 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:34.205539 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:34.210163 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:34.377496 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:34.399363 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:34.422017 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:34.436177 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:34.437230 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:34.438150 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:34.438613 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:34.473714 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:27:34.612605 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:34.612702 15916 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:27:34.612885 15916 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:27:34.613538 15916 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:27:34.614567 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:27:34.615164 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:34.615351 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:27:34.615622 15916 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:27:34.689358 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:34.744612 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:34.885815 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:34.897869 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:34.908367 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:34.918661 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:34.922488 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:34.940021 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:35.084151 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:35.152323 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:27:35.174067 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:35.180979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:35.193292 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:35.197320 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:35.198117 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:35.396385 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:35.408375 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:35.420444 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:35.424997 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:35.426187 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:35.430063 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:35.431867 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:35.455790 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:35.467736 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:35.549818 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:35.559381 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:35.562521 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 08:27:35.616451 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:35.678391 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:35.678527 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:35.688063 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:35.689383 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:35.718509 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:35.778735 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:35.897784 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:35.980254 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:36.016498 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:27:36.021438 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:36.114924 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:36.116137 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:36.122283 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:36.123315 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:36.130949 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:36.143697 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:36.151189 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:36.190246 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:36.364020 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:36.368247 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:36.373433 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:36.390230 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:36.399931 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:36.414407 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:36.469177 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:36.604327 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:36.620739 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:36.636806 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:36.639655 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:36.655716 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:36.683617 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:36.757104 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:36.846580 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:27:36.864552 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:36.941720 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:36.942572 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:36.942806 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:36.954542 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:36.957114 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:36.991622 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:27:37.127377 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:37.200713 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:37.228574 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:37.248728 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:37.376945 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:37.394145 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:37.394929 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:37.396970 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:37.412312 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:37.588613 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:27:37.646312 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:37.652325 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:37.667588 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:37.669044 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:37.670509 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:37.672802 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:37.813961 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:37.904574 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:37.907799 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:37.947556 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:37.950274 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:37.951769 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:37.961340 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:37.963667 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:37.987141 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:38.118304 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:38.177371 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:38.193578 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:38.199483 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:38.205809 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:38.211150 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:38.236840 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:38.258201 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:38.273643 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:38.440208 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:38.446557 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:38.471920 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:38.510648 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:38.518617 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:38.617007 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:38.634121 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:38.646982 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:38.653211 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:38.666674 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:38.741536 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:38.806100 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:38.911698 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:38.918726 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:38.920393 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:38.919502 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:27:38.950160 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:38.951324 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:39.089452 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:39.167395 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:39.200002 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:39.219843 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:39.227062 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:39.231159 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:39.234633 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:39.314759 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:39.476303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:39.497900 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:39.498154 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:39.498900 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:39.499792 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:39.504215 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:27:39.520373 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:39.524618 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:39.534394 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:39.700817 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:39.703204 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:39.877980 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:39.917646 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:39.919092 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:39.933123 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:39.982602 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:40.034108 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:40.186087 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:27:40.214628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:40.215901 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:40.244707 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:40.247656 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:40.253878 21488 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 47865.9us
I20240502 08:27:40.273541 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:40.317827 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:40.319278 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:40.474368 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:40.479463 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:40.483772 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:40.484161 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:40.519356 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:40.527007 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:40.527226 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:40.647454 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:27:40.698303 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:40.711086 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:40.716892 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:27:40.723429 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:40.730059 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:40.736280 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:40.801221 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:40.804857 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:40.932662 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:41.022678 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:41.048736 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:41.090106 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:41.180034 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:41.182386 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:41.201239 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:41.216256 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:41.217466 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:41.254174 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:41.307212 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:41.308197 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:41.344323 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:41.417335 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:41.459308 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:41.460505 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:41.467167 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:41.488914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:41.496457 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:41.502233 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:41.705080 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:41.726174 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:27:41.732053 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:41.732086 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:41.733883 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:41.757419 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:41.769850 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:41.850198 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:41.897158 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:41.922638 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:41.924296 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:27:41.925328 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:42.002535 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:42.003468 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.015260 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:42.026227 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:42.026974 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.040246 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:42.215454 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:42.243638 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:42.444777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.451925 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:42.471947 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:42.472009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.478787 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:42.618541 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:27:42.667536 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.680855 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:42.686525 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:42.699419 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:42.705677 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:42.841898 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:42.942162 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:42.955437 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:42.960943 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:42.962577 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:42.966918 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:42.996574 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:27:43.088516 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:43.095005 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.101550 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.167109 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:43.183992 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:43.199812 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:43.214395 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:43.222836 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:43.231566 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:27:43.243726 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:43.345660 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.396019 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:43.458400 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:43.592381 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:43.651963 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:43.665899 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:43.673774 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:43.683795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:43.687959 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:43.719650 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.777487 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.778091 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:43.918936 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:43.954077 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:43.959131 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:43.965265 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:43.973193 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:43.973794 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:44.023447 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:44.091172 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:44.148622 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:44.170845 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:44.176596 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:44.178074 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:44.189426 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:44.201086 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:44.205570 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:44.236881 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:44.295419 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:44.405253 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:44.426306 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:44.429083 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:44.441978 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:44.456902 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:44.456982 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:44.483641 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:27:44.661851 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:44.770421 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:44.829905 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:27:44.872843 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:44.882227 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:44.888695 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:44.911043 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 49515.3us
W20240502 08:27:44.913319 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:44.914939 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:44.937948 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:45.104665 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:45.117777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:45.124591 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:45.140194 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:45.148288 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:45.148994 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:45.318351 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:45.373505 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:45.376549 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:45.401433 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:45.401598 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:45.419047 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:27:45.439745 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:45.630506 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:45.674314 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:27:45.677212 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:45.689848 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:45.691089 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:45.699276 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:45.700695 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:45.713361 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:45.758301 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:45.759696 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:45.841168 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:46.025904 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:46.098248 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:46.118600 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:46.152068 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:46.155576 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:46.161505 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:46.180001 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:46.182529 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:27:46.183511 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:46.186276 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:46.187003 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:46.284251 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:46.310257 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:46.402765 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:46.413646 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:46.418605 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:46.426955 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:46.433393 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:46.482604 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:46.628718 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:46.636034 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:46.641369 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:46.646277 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:27:46.656268 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:46.656551 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:46.659320 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:46.716353 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:46.730736 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:46.766631 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:46.842350 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:46.895299 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:46.936497 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:46.948601 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:46.950536 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:46.971688 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:46.988451 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:47.040210 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:27:47.059836 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:47.241418 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:47.408006 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:47.420410 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:47.424597 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:47.429924 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:47.435565 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:47.646598 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:47.649372 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:47.656167 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:47.658756 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:47.661685 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:47.662169 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:47.768865 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:47.835881 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:47.870693 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:47.889256 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:47.915647 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:47.918071 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:47.926406 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:47.998622 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:27:48.056012 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:48.134703 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:27:48.142751 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.152135 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:48.165243 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:48.168406 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.172421 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:48.341202 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:48.379346 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:48.381350 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:48.507278 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:48.559291 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:48.578804 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.588531 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:48.590075 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:48.604775 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:48.605383 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:48.607513 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.615589 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:48.653337 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:48.804478 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:48.844157 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:48.844338 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.868745 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:48.874087 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:48.908556 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:48.916381 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:48.925457 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:49.042639 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:49.073931 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:49.136835 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:49.146802 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:49.151510 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:49.169011 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:27:49.186870 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:49.188674 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:49.200825 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:49.324067 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:49.398564 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:49.413450 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:49.424679 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:49.450822 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:49.452538 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:49.454409 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:49.556031 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:49.628330 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:49.757728 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:49.897910 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:49.908834 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:49.912089 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:49.919454 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:49.920701 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:49.921168 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:49.929605 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:27:50.014741 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:50.137851 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:50.138298 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:50.138679 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:50.168339 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:50.173166 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:50.209982 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:50.386052 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:50.386680 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:50.392447 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:50.407994 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:50.411458 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:50.489604 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:50.511734 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:27:50.591152 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:50.628562 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:50.636243 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:50.636695 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:50.640231 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:50.643112 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:50.698738 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:27:50.874603 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:50.910992 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:50.953673 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:50.960742 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:51.006279 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:51.063541 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:51.065299 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:51.077279 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:51.081010 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:51.082026 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:51.109995 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:51.155022 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:51.300827 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:51.317188 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:51.338387 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:51.351320 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:51.359812 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:51.361167 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:51.398303 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:51.414166 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:51.448168 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:51.467401 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:51.503367 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:51.517138 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:51.557984 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:51.570735 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:51.574721 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:51.575158 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:51.591058 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:51.607921 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:51.626390 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:27:51.746846 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:51.868103 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:51.889950 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:51.901295 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:51.902693 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:51.907768 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:51.912901 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:51.989109 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:52.116544 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:27:52.193601 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:52.272329 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:52.279484 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:52.331202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:52.361867 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:52.388136 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:52.398094 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:52.417431 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:52.446382 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:52.590831 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:27:52.595480 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:52.608722 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:52.610742 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:52.611656 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:52.638211 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:52.776293 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:52.849089 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:52.873319 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:52.909868 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:52.910830 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:52.921200 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:53.042567 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:27:53.152762 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:53.154047 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:53.155601 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:53.162041 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:53.163269 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:53.168515 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:53.187501 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:53.225134 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:53.465788 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:53.490732 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:53.527092 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:53.568209 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:27:53.619606 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:53.621318 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:53.627038 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:53.627424 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:53.636706 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:53.647508 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:53.652997 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:53.676028 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:53.707554 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:53.846223 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:53.874409 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:53.875442 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:53.912547 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:53.918898 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:53.937791 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:53.954239 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:27:54.068329 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:54.073185 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:54.112605 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:54.125340 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:54.129148 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:54.138248 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:54.138393 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:54.147614 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:54.151325 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:54.170516 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:54.198406 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:54.265434 22627 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:27:54.265677 22627 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:27:54.266731 22627 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:27:54.268306 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:27:54.268500 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:54.268697 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:27:54.268965 22627 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:27:54.363088 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:54.402500 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:54.405090 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:54.418653 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:54.423835 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:54.424700 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:54.493419 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:54.625264 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:54.770260 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:54.811118 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:54.851269 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:54.859107 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:54.862852 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:54.877393 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:54.927714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:54.941069 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:55.105638 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:55.110168 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:55.122573 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:55.124552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:55.136368 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:55.141342 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:55.160528 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:27:55.340330 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:55.376610 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:55.379325 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:55.380348 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:27:55.428786 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:55.429739 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:55.542325 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:27:55.634995 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:55.636034 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:55.642639 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:55.662799 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:55.686065 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:55.726361 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:55.736724 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:27:55.747687 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:55.853089 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:55.944360 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:56.051216 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:56.081748 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:56.103500 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:56.104965 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:56.157338 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:56.163350 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:56.167708 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:56.168490 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:56.191864 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:56.243939 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:56.263075 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:56.315179 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:56.349875 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:56.388765 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:27:56.396618 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:56.398293 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:56.422554 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:56.424680 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:56.443459 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:56.534245 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:56.611878 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:56.624490 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:56.635617 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:56.645085 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:56.665833 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:56.695399 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:27:56.698184 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:56.738042 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:56.746275 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:27:56.777249 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:56.873346 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:27:56.901506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:56.924952 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:56.926504 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:56.928012 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:56.934194 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:57.203933 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:57.249703 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:27:57.269315 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:57.311468 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:57.344738 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:57.369525 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:57.375309 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:57.375725 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:57.396510 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:57.415756 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:27:57.567003 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:57.567760 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:57.571878 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:57.595465 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:57.618161 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:57.686292 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:57.835376 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:57.842317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:57.844256 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:57.851348 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:57.869307 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:57.876986 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:58.083304 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:58.084405 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:58.099422 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:58.103178 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:58.112886 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:58.130235 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:58.151063 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:58.166821 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:27:58.228060 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:27:58.228991 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:27:58.282522 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:58.386024 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:27:58.467280 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:27:58.547068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:58.563936 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:58.568789 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:58.575644 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:58.584388 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:58.656265 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:27:58.680294 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:27:58.695449 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:27:58.775400 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:27:58.801479 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:58.822000 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:58.843688 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:27:58.846298 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:58.849115 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:58.883666 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:58.919785 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:58.920266 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:58.978937 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:59.115926 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:59.119954 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:59.127071 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:59.131983 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:59.144344 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:27:59.158453 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:59.329499 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:27:59.374279 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:59.398892 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:27:59.400328 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:59.404922 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:27:59.408658 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:27:59.415053 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:27:59.425647 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:27:59.472749 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:27:59.631662 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:27:59.842118 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:59.892922 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:27:59.897130 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:27:59.915552 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:27:59.928555 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:27:59.987766 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:00.042169 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:28:00.063778 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:28:00.109498 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:00.112861 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:00.114771 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:00.146494 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:00.153666 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:00.367384 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:28:00.375571 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:00.384469 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:00.386008 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:00.386430 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:00.389693 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:00.418859 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:00.608736 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:00.621368 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:00.623764 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:00.628839 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:00.632592 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:00.637940 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:00.648663 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:28:00.710866 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:00.745591 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:28:00.746501 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:00.869235 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:28:00.877454 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:28:00.896335 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:01.072084 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:01.081035 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:01.084475 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:01.102005 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:01.107388 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:01.166546 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:01.186303 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:28:01.218212 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:01.234282 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:01.303154 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:28:01.342346 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:01.346916 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:01.367442 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:28:01.368778 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:01.371886 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:01.373272 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:01.377159 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:01.407656 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:01.467196 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:28:01.587692 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:01.588889 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:01.593101 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:01.597151 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:01.603955 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:01.708046 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:01.819865 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:01.833357 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:01.856068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:01.862131 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:01.872609 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:01.889374 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:01.905584 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:01.917901 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:01.927311 25582 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:28:01.927542 25582 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:28:01.928283 25582 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:28:01.930748 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:28:01.931058 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:01.931165 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:28:01.931869 25582 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:28:01.999711 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:02.125147 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:02.266719 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:02.300421 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 08:28:02.389452 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:02.404815 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:02.405378 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:02.406159 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:02.409029 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:02.454675 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:02.611495 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:28:02.667199 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:02.675539 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:02.679631 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:02.700244 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:02.705853 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:02.731009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:02.899798 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:28:02.932118 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:02.935045 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:02.944563 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:02.957584 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:02.958984 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:03.052047 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:03.095880 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:03.184633 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:03.196501 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:03.209996 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:03.232713 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:03.232982 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:03.233178 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:03.281615 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:03.308202 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:03.327873 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:03.402386 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:28:03.479442 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:03.479470 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:03.509994 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:03.562888 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:03.628180 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:03.642853 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:03.655093 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:03.658358 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:03.664834 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:03.680441 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:03.680691 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:03.693289 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:03.771054 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:03.792863 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:28:03.854826 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:28:03.885118 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:03.902730 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:03.932173 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:03.956288 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:03.973366 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:04.058883 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:04.149658 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:04.152201 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:28:04.156767 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:04.159230 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:04.169348 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:04.184238 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:04.417376 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:04.420300 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:04.437755 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:04.444741 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:04.456290 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:04.476060 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:04.541283 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:04.660506 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:04.753897 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:04.889365 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:04.923933 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:04.924671 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:04.927857 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:04.936604 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:04.950054 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:05.019997 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:05.092610 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:28:05.132921 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:05.137740 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:05.138972 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:05.146013 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:05.149469 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:05.360458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:05.370785 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:05.388015 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:05.393236 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:05.393956 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:05.414942 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:05.473402 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:05.533425 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:05.579380 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:05.596972 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:05.624387 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:05.627317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:05.633812 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:05.643734 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:05.767357 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:05.769742 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:28:05.841970 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:05.860335 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:05.887502 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:05.931874 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:05.944298 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:05.947376 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:05.979197 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:05.990147 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:28:06.048678 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.053117 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:06.054093 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:06.076481 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:06.093937 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:06.124965 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:06.176761 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:28:06.299217 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:06.306540 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.311527 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:06.325232 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:06.344415 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:06.410880 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:06.428288 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:06.461208 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:06.521435 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:06.522205 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:06.589411 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.602384 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:06.606290 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:06.609236 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:06.617560 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.721194 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033345
I20240502 08:28:06.726871 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.000s Metrics: {"bytes_written":2352,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":219,"threads_started":1,"update_count":40,"wal-append.queue_time_us":323}
I20240502 08:28:06.727483 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.159005
I20240502 08:28:06.740561 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15279,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":595,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":122,"lbm_read_time_us":269,"lbm_reads_lt_1ms":17,"lbm_write_time_us":273,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":490,"threads_started":2,"update_count":40}
I20240502 08:28:06.871435 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:06.871944 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.892685 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:06.904582 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:06.937038 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:06.997766 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:07.095557 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:07.258661 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:07.301254 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:07.328923 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:28:07.339195 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:07.344174 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:07.367390 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:07.373147 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:28:07.437717 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:07.462409 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:07.564461 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:07.567504 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:07.629841 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:07.644243 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:07.650063 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:07.658660 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:07.669301 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:07.764577 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:07.877794 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:07.881987 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:07.890489 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:07.908560 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:07.909708 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:08.076663 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.086134 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.086838 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:08.116284 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:08.132071 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:08.136734 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:08.158545 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:08.160087 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:08.179898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:08.276717 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.320917 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.410910 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:08.428180 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:08.438088 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:28:08.459887 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:28:08.517182 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:08.522804 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.549377 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:08.575937 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:08.592849 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:08.601214 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:08.620122 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:08.629531 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:08.696628 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:08.834084 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:28:08.836151 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:08.843698 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:08.871378 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:08.890200 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:08.913583 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:08.934216 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:08.973037 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:08.986109 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:09.006495 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:09.135349 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:09.145923 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:09.150617 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:09.153018 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:09.176138 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:09.182726 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:09.368763 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:09.380736 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:09.385773 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:09.414829 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:09.416878 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:09.417999 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:09.654882 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:09.750464 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:09.815194 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:09.849817 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:09.891618 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:09.895792 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:09.915903 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:09.916922 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:09.943075 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:09.959486 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 49109.9us
I20240502 08:28:10.171133 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:10.173063 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:10.192170 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:10.197829 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:10.208856 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:10.283211 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:10.290421 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:10.309496 25582 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:28:10.309705 25582 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:28:10.310479 25582 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:28:10.312458 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:10.312937 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:28:10.313042 23406 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:28:10.313283 25582 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:28:10.334331 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:10.422243 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:10.427788 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:10.460143 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:10.473527 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:10.490597 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:10.505801 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:10.530599 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:10.590981 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:10.603425 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:28:10.694976 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:10.704340 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:10.713430 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:10.734843 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:10.739311 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:10.750574 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:10.773327 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:10.778384 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:10.849741 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:28:10.912887 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:11.001171 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:11.014001 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:28:11.052420 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:11.090960 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:11.151816 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:11.160257 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:11.181345 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:11.184640 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:11.186256 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:11.187217 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:11.250303 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:11.337328 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:28:11.386197 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:11.393041 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:11.403467 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:28:11.412523 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:11.428085 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:11.432133 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:11.435729 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:11.465296 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:11.580782 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:28:11.610221 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:11.619047 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:11.628561 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:11.632246 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:11.633401 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:11.643826 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:11.886248 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:11.888520 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:11.890683 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:11.892488 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:11.910795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:12.012802 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:12.035392 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:12.129479 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:12.336339 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:12.353135 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:12.371712 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:12.382706 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 08:28:12.388257 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:12.439718 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:12.445484 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:12.497593 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:28:12.498893 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:28:12.637993 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:12.646922 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:12.648983 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:12.662323 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:12.666921 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:12.694643 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:12.718748 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:12.897871 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:12.913803 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:12.918081 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:12.925197 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:12.967664 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:12.980654 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:28:12.981957 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:12.982108 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:28:13.010550 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:13.029760 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:13.096786 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:28:13.161036 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:13.161021 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:13.162533 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:13.180122 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:13.192178 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:13.205945 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:13.245761 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:13.262931 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:13.417178 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:13.487135 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:13.500808 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:13.551167 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:28:13.583151 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:13.598711 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:13.624591 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:13.625156 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:13.627635 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:13.650130 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:13.651695 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:13.669358 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:13.689966 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:13.786264 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:13.887694 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:13.903805 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:13.923487 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:13.923822 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:13.924969 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:13.935227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:14.062323 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:14.088706 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:28:14.121148 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:14.132535 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:14.142539 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:14.150329 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:14.152065 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:14.154896 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:14.175493 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:14.331208 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:28:14.400885 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:14.433717 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:14.445613 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:14.451268 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:14.465288 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:14.490885 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:14.859891 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:14.911793 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:14.935365 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:14.941450 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:14.951598 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:14.953158 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:15.015312 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:15.050029 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:15.084250 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:15.176364 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:15.187033 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:15.195649 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:15.209851 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:15.210378 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:15.359526 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:15.383404 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:15.418244 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:15.431844 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:15.449343 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:15.451352 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:15.452577 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:15.514421 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:15.578712 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:15.599169 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:15.676142 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:15.684823 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:15.687970 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:15.692451 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:15.700124 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:15.755945 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:15.761806 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:15.789698 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:15.867987 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:15.868952 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:15.896567 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:28:16.003808 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:16.022521 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:16.105302 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.111222 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:16.119648 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:16.140082 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.147972 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:16.170205 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:16.184366 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:16.272292 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:16.312247 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:16.314951 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:16.368152 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.377355 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:16.385964 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:16.401132 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:16.406711 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:16.468194 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:16.545116 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:16.546154 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:28:16.579457 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:16.586521 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.596740 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:16.598258 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:16.601155 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:16.628744 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.875286 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:16.888816 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:16.892288 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:16.893702 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:16.924751 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:16.980212 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:17.148522 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:17.363428 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:17.408584 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:17.411208 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:28:17.412840 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:17.413836 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:17.416451 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:17.417608 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:17.444126 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:17.546350 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:17.609711 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:17.621068 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:17.622298 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:17.626572 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:17.635717 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:17.653601 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:17.765228 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:17.801370 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:17.876427 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:17.879683 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:17.884615 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:17.909541 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:17.944943 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:17.957131 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:17.986331 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:18.101465 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:18.131009 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:18.154979 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:18.156613 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:18.157317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:18.158838 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:18.218509 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:18.224503 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:18.260121 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:18.272001 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:18.280393 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:18.450383 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:28:18.459357 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:18.570628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:18.571923 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:18.574137 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:18.584224 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:18.601078 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:18.650658 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:18.663480 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:28:18.739622 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:18.773450 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:18.822576 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:18.826496 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:18.839704 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:18.853119 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:28:18.859102 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:18.864096 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:18.901219 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:18.906399 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:18.951817 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:18.952675 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:19.050336 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:19.086033 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:19.087775 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:19.090163 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:19.104099 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:19.111204 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:19.310588 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:19.310765 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:19.338011 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:19.339563 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:19.346961 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:19.382475 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:19.578758 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:19.733675 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:19.762202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:19.766002 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:19.774317 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:19.793601 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:19.799882 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:19.816367 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:19.963894 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:20.006146 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:20.045353 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:20.045928 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:20.050966 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:20.051537 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:20.074856 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:20.261334 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:20.272946 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:20.278864 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:20.295006 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:20.321009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:20.395982 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:20.450230 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:20.525538 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:20.525929 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:20.527395 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:20.542830 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:20.548976 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:20.549549 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:20.580130 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:20.580117 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:20.598084 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:20.604851 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:20.700223 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:20.785331 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:28:20.796581 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:28:20.969798 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:20.994920 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:20.996259 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:21.029536 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:21.029990 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:21.079869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:21.118654 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:21.167538 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 08:28:21.209182 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:21.221411 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:21.229539 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:21.249049 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:21.262126 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:21.263646 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:21.283648 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:21.288369 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:21.386138 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:21.390393 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:21.495924 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:21.509582 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:21.524106 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:21.527011 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:28:21.531106 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:21.537014 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:21.597462 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:21.597613 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:21.694463 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:21.758503 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:21.762619 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:21.775457 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:21.777664 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:21.780404 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:21.800698 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:22.028262 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:22.131594 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:22.216889 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:22.236336 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:22.237699 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:22.240638 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:22.272626 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:22.313817 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:22.429208 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 08:28:22.496594 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:22.509438 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:22.516605 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:22.520674 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:22.538609 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:22.658481 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:22.703588 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:22.721083 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:22.731561 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:22.756316 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:22.762769 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:22.786701 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:22.926815 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:22.968091 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:22.991816 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:23.047803 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:23.048301 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:23.072741 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:23.075102 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:23.093883 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:23.095429 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:23.095897 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:23.119288 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:23.197934 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:23.308214 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:23.319312 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:23.322613 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:23.371281 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:23.402014 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:23.410454 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:23.482409 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:23.504822 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:23.512935 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:23.516721 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:23.519479 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:23.575556 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:28:23.716367 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:23.731814 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:23.732355 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:23.737097 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:23.755736 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:23.774356 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:23.829193 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:28:23.830317 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:23.884442 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:23.934451 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:23.975600 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:23.977797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:23.992272 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:28:23.997561 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:24.005826 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:24.011718 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:24.025094 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:24.211210 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:24.229701 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:24.235035 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:24.254853 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:24.255498 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:24.261713 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:24.264317 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:24.669752 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:24.670215 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:24.681869 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:24.682139 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:24.694474 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:24.754618 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:24.801380 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:24.866380 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:24.952024 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:24.976946 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:24.980213 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:24.981238 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:24.984177 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:25.052112 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:25.200491 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:25.211118 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:25.225270 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:25.226758 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:25.241339 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:25.352880 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:25.422626 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:25.424748 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:25.463315 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:25.483538 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:25.499269 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:25.499819 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:25.502388 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:25.530691 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:25.649273 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:25.695428 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:25.732046 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:25.771023 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:25.775521 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:25.960215 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:25.962558 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:25.964926 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:25.967798 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:25.968237 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:25.971215 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:26.004272 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:26.006366 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:26.073114 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:28:26.108057 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:26.182582 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:26.197911 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:26.200143 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:26.204555 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:26.205132 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:26.209146 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:28:26.210081 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:28:26.341071 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:26.389986 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:28:26.403476 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:26.414726 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:26.417841 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:26.421480 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:26.447674 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:26.467279 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:26.580186 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:26.583334 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:26.647594 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:26.661759 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:26.666281 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:26.687783 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:26.695806 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:26.800442 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:26.842242 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:27.016994 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:27.068792 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:27.093888 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:27.096997 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:27.106343 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:27.116053 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:27.183591 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:27.350777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:27.361088 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:27.379781 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:27.379802 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:27.419806 25582 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:28:27.420012 25582 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:28:27.420903 25582 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:28:27.421748 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:28:27.428033 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:27.436007 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:27.436321 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:28:27.436678 25582 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:28:27.441641 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:27.474484 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:28:27.477710 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:27.489709 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:27.616614 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:27.624141 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:27.626828 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:27.627549 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:27.690795 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:27.779091 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:27.867017 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:27.875146 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:27.880298 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:27.880997 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:27.890914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:27.915094 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:27.953400 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:28.044214 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:28.084192 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:28.300926 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:28.354480 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:28.357602 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:28.371832 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:28.373991 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:28.374213 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:28.384030 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:28.391165 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:28.416952 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:28.453734 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:28.485420 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:28.490515 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:28.571815 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:28:28.593156 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:28.600332 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:28.606524 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:28.619554 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:28.620747 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:28.633105 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:28:28.635991 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:28.773713 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:28:28.825755 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:28.849618 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:28.862489 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:28.875569 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:28:28.889562 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:28.891630 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:28.905431 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:28.917860 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:28.970974 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:28.973418 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:28.975106 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:28:29.110718 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:29.111488 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:29.119463 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:29.120150 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:29.129746 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:29.148993 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:29.306713 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:29.527181 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:29.540225 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:29.541293 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:29.560086 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:29.567970 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:29.722553 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:28:29.757050 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:29.761770 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:29.778889 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:29.789750 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:29.793185 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:29.847790 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:29.859724 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:29.862675 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:28:30.048830 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:30.056519 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:30.080531 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:30.081995 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:30.088930 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:30.198711 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:30.239192 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:30.269776 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:30.313068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:30.329255 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:30.334124 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:30.335183 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:30.390722 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:30.624915 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:30.644142 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:30.678014 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:30.722220 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:30.742900 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:30.748925 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:30.798898 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:30.817978 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:30.822408 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:30.824335 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:30.847133 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:30.848083 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.075044 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.084163 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:31.084870 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:31.105815 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.118785 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:31.127797 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:28:31.136459 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:31.147419 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:31.150507 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:31.212553 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:31.213485 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:28:31.297554 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:31.319401 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.326778 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:31.328182 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:31.332437 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:31.346493 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:31.368458 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:31.403628 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:31.472113 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:31.552860 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.578689 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:31.580538 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.581459 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:31.584062 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:31.600195 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:31.680655 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:31.826153 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:31.848729 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:28:31.983223 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:31.988230 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:31.999282 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:32.013751 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:32.025871 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:32.100298 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:32.195278 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:32.219296 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:32.220850 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:32.224121 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:32.231528 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:32.247162 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:32.259678 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:32.262964 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:32.432655 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:32.443097 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:32.453660 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:32.471627 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:32.480074 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:32.563014 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:28:32.632441 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:32.705004 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:32.712165 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:32.722469 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:32.738874 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:32.738916 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:32.825893 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:32.828083 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:33.152628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:33.177595 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:33.196072 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:33.209355 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:33.228886 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:33.240418 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:33.264618 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:33.300621 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:33.306288 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:33.324350 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:33.331399 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:33.449484 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:33.451196 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:33.452826 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:33.456284 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:33.476524 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:33.497102 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:33.515048 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:33.595217 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:33.635677 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:33.659715 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:33.662420 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:33.669885 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:33.676010 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:28:33.676899 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:33.679313 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:33.679894 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:33.697008 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:28:33.698623 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:28:33.699465 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:33.881372 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:33.882797 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:33.927192 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:33.931861 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:33.933010 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:33.942940 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:33.957823 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:33.959965 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:34.101946 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:34.145884 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:28:34.359726 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:34.372524 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:34.385696 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:34.399135 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:34.439196 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:34.451089 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:34.625489 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:34.627928 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:34.631549 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:34.638132 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:34.649974 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:34.729876 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:34.744663 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:34.774006 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:34.842317 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:34.862542 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:34.863185 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:34.880445 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:34.883770 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:34.928345 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:34.980052 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:35.115948 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:35.131150 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:35.141892 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:35.143007 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:35.155293 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:35.313263 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:35.370668 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:35.538098 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:35.565457 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:35.574926 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:35.577512 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:35.583685 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:35.698055 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:35.734998 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:35.761391 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:35.785032 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:35.809751 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:35.839704 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:35.865489 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:35.884616 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:35.890602 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:35.911336 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:36.010176 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:36.016690 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:36.046873 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:36.060539 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:36.091349 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:36.091815 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:36.105551 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:36.108222 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:36.108412 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:36.111722 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:36.126775 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:36.154045 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:36.271351 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:36.275002 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:36.281219 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:28:36.331156 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:36.357100 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:36.371865 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:36.374045 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:36.374724 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:36.416585 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:36.508550 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:36.734156 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:36.834663 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:36.841137 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:36.855693 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:36.869149 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:36.893796 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:36.897562 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:36.940413 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:37.109233 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:37.112238 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:37.114790 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:37.129570 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:37.144361 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:37.276631 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:37.332604 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:37.343765 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:37.347162 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:37.379570 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:37.385986 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:37.387591 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:37.424445 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:37.428133 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:37.480353 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:37.583703 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:28:37.615468 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:37.624789 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:37.625547 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:37.628437 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:37.647553 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:37.691526 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:37.973683 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:38.049190 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:38.065727 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:38.067328 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:38.068565 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:38.072111 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:38.072129 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:38.089637 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:38.093340 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:38.229221 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:38.275548 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:38.276530 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:38.283567 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:38.310536 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:38.342646 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:38.433409 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:38.458436 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:38.524628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:38.527472 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:38.537102 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:38.539471 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:38.554246 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:28:38.563136 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:38.563952 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:38.567910 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:38.604969 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:38.626549 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:38.678948 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:38.720237 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:28:38.762908 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:38.765580 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:38.782189 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:38.785681 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:38.790040 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:28:38.812570 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:38.827952 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:38.957990 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 08:28:38.958876 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:39.006839 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:39.025035 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:39.209290 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:39.226553 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:39.234727 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:39.249871 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:39.260879 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:39.285262 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:39.359539 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:39.414685 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:39.497243 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:39.518226 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:39.526979 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:39.531569 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:39.544731 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:39.574165 21487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 60956.7us
I20240502 08:28:39.733954 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:39.734095 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:39.734661 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:39.738752 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:39.748581 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:39.795811 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:39.799331 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:39.917635 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:39.966619 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:39.980410 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:39.980525 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:39.985780 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:39.995452 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:40.123677 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:40.180462 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:40.313025 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:40.427634 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:40.440577 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:40.443320 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:40.444770 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:40.472113 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:40.592176 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:40.601410 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:40.615720 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:40.645601 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:40.659852 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:40.672183 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:40.672224 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:40.673143 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:40.724908 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:40.894501 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:40.926115 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:40.938578 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:40.955730 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:40.955735 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:40.968073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:41.092185 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:41.131031 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:41.131114 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:41.157593 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:41.160627 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:41.178715 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:41.183591 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:41.187970 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:41.193636 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:41.203867 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:41.207042 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:28:41.213335 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:41.272416 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:28:41.361534 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:41.392956 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:41.397475 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:41.473913 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:28:41.612866 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:41.625293 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:41.625293 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:41.627485 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:41.636186 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:41.648995 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:41.668035 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:28:41.828312 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:41.852507 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:41.858678 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:41.862253 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:41.883966 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:41.915627 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:42.092545 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:42.099435 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:42.108250 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:42.109218 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:42.115417 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:42.150427 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:42.252516 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:42.304440 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:42.313243 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:42.315618 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:42.327785 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:42.347803 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:42.388742 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:42.404832 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:42.501351 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:42.629549 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:28:42.716372 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:42.746497 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:42.758512 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:42.762598 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:42.764669 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:42.795959 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:42.807375 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.016587 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:43.016997 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.030117 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:43.035437 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:43.048756 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:43.093537 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:43.148964 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:43.184975 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:43.213516 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:43.235019 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.249964 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:43.250504 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:43.254547 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:43.257951 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:43.327826 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:43.444722 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:43.452164 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:43.508364 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:43.523113 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.530599 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:43.531211 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:43.548656 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.552927 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:43.596943 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:43.620254 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:43.644383 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:43.681536 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:43.688297 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:43.740546 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:28:43.802937 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:43.927062 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:43.938982 30814 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:28:43.939191 30814 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:28:43.940245 30814 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:28:43.941606 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:28:43.942601 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:43.942751 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:28:43.943027 30814 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:28:43.959805 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:28:43.961095 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:43.971498 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:43.971598 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:43.993474 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:44.000398 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:44.022675 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:44.164685 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:44.219800 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:44.227088 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:44.237442 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:44.249145 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:44.252729 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:44.265997 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:44.288303 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 87520us
I20240502 08:28:44.298147 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:44.316890 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:44.473384 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:44.480393 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:44.480609 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:44.489925 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:44.490666 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:44.690347 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:44.706744 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:44.711340 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:44.712476 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:44.714653 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:44.760541 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:44.768553 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:44.975597 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:45.055531 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:45.063158 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:45.132086 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.133460 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:45.152881 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:45.157996 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.163573 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:45.299496 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:45.411269 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.417418 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:45.422854 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:45.426348 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:45.441792 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:45.520145 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:45.633211 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.647536 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:45.653434 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:45.658880 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.679762 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:45.754616 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:45.847211 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:45.877188 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:45.890827 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:28:45.898826 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:45.902318 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:45.910483 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:45.913414 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:45.949043 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:45.991109 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:46.053800 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:46.067312 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:46.168298 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:46.211288 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:28:46.266376 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:46.323626 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:46.342455 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:28:46.381577 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.389885 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:46.398249 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.399555 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:46.423549 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:46.426400 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:46.550608 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:46.615087 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:46.640961 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.658850 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.662379 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:46.664940 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:46.665102 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:46.680925 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:28:46.754518 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:46.811358 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:46.871052 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.881438 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:46.893322 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:46.910846 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:46.911401 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:46.914904 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:47.096350 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:47.103945 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:47.125115 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:47.128919 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:47.137133 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:47.160604 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:47.165366 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:47.401852 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:47.438359 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:47.508162 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:47.569895 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:47.574285 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:47.591704 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:47.592329 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:47.595137 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:47.648350 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:47.746840 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240502 08:28:47.808213 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:47.813751 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:47.832556 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:47.835402 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:47.845185 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:47.847028 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:47.895366 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:47.957443 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:48.002770 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:48.089993 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:48.100718 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:48.108898 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:48.120251 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:48.134009 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:48.219709 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:48.328986 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:48.346974 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:48.355176 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:48.356609 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:48.397900 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:48.413209 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:48.416725 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:48.429320 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:48.478205 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:48.526691 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:48.606395 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:28:48.656463 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:28:48.742497 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:48.779826 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:48.854089 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:48.863932 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:48.877740 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:48.889125 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:48.892979 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:48.898217 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:48.899114 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:48.920902 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:49.044564 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:28:49.105427 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:49.110092 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:49.130686 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:49.136842 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:49.157039 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:49.159301 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:49.261086 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:49.299312 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:28:49.333706 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:49.375679 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:49.387974 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:49.388689 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:49.397001 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:49.418128 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:28:49.420006 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:49.453778 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:49.494565 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:28:49.630673 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:49.632447 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:49.650453 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:49.653286 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:49.655071 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:49.666419 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:49.971086 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:50.046178 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:50.054958 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:50.067996 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:50.071553 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:50.086053 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:50.104532 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:50.274560 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:28:50.296464 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:50.314410 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:50.321434 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:50.325932 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:50.356783 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:50.424222 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:50.549351 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:50.567621 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:50.570437 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:50.574554 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:50.590453 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:50.590858 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:50.608382 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:50.692335 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:28:50.835510 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:50.850692 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:50.863570 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:50.875490 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:50.880367 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:50.971395 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:51.021840 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:51.056118 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:51.144912 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:51.164158 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:51.195328 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:51.235116 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:51.287204 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:51.307277 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:28:51.326364 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:51.337463 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:51.342877 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:51.347672 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:51.376451 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:51.401536 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:51.525620 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:51.612378 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:51.615547 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:51.627413 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:51.631640 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:51.633021 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:51.652172 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:51.664750 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:51.700449 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:28:51.887219 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:51.908396 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:51.912999 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:51.913704 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:51.914340 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:51.952701 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:51.971653 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:52.138010 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:52.138520 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:52.139181 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:52.150020 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:52.164206 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:52.174611 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:52.174571 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:52.210698 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:52.232192 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:52.571024 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:52.577708 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:52.583284 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:52.597688 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:52.604573 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:52.605542 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:52.605973 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:52.632256 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:52.656466 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:52.769637 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:28:52.815537 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:52.820122 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:52.841145 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:52.856640 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:52.882782 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:53.021723 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:28:53.075623 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:53.083182 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:53.130074 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:53.133966 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:53.134310 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:53.143080 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:53.203809 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:53.245267 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:53.273396 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:53.381486 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:53.403911 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:53.412277 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:53.412716 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:53.413205 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:53.419981 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:53.454244 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:53.457389 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:53.599615 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:53.663126 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:53.790699 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:53.844238 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:53.846668 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:53.853082 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:53.879065 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:53.891441 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:53.929011 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:53.948788 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:53.956684 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:54.013501 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:28:54.072156 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 08:28:54.093628 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:54.104139 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:54.123760 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:54.124275 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:54.125141 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:54.204183 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:54.252840 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:54.354772 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:54.364813 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:54.365614 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:54.392470 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:54.421618 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:54.469842 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:54.596408 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:54.600137 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:54.600790 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:54.606873 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:54.620529 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:28:54.620693 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:54.687615 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:54.687889 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:28:54.689215 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:28:54.697178 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:54.751001 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:55.020869 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.027223 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:55.028542 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:55.033897 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.035914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:55.045056 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:55.089223 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:55.198417 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:55.238173 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:55.244285 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:55.249897 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:55.262312 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:55.283372 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:55.287761 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.468010 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.495649 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:55.495872 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:55.509770 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:55.510149 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:55.612790 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:55.657879 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:55.720162 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:55.726439 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.739784 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:55.753999 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:55.761590 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:55.769266 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:55.770493 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:55.946667 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:28:55.965739 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:55.995539 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:56.004441 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:56.220785 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:56.222249 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:56.234716 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:56.238359 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:56.238411 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:56.248265 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:56.292294 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:56.404105 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:28:56.407582 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:56.452193 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:56.473438 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:56.499440 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:56.500059 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:28:56.500252 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:56.507694 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:56.532032 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:56.570681 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:56.624378 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:28:56.692926 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:56.693241 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:56.726392 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:56.727947 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:56.771034 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:56.819993 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:28:56.852648 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:28:56.916872 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:56.928879 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:56.959748 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:56.963292 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:56.972281 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:56.973899 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:56.990741 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:57.005641 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:57.350697 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:57.350729 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:57.407112 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:57.415323 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:57.434871 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:57.437565 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:57.439195 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:28:57.463445 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:57.547288 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:57.647958 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:57.653376 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:57.670928 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:57.676668 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:57.677166 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:57.767194 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:57.784427 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:28:57.785509 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:57.850615 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:28:57.898186 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:57.905179 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:57.924640 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:57.931082 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:57.953665 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:58.061734 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:28:58.121121 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:58.152257 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:58.160449 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:58.161275 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:58.177085 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:58.186981 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:58.259748 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:28:58.280722 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:58.304462 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:58.353598 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:58.493151 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:28:58.598726 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:58.614596 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:58.617152 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:58.617983 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:58.638952 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:58.654223 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:58.715343 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:28:58.741640 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:58.800545 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:58.821832 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:28:58.850417 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:58.872684 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:58.879753 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:58.892753 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:58.896745 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:59.002405 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:28:59.019493 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:28:59.052006 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:28:59.063612 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:28:59.092043 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:28:59.104195 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:28:59.127113 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:59.131345 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:59.132231 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:59.141366 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:59.158926 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:28:59.180079 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:59.268535 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:28:59.355540 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:28:59.372038 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:59.372206 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:59.404932 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:59.428437 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:59.431104 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:28:59.669191 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:28:59.771533 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:28:59.916215 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:28:59.928144 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:28:59.930493 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:28:59.931023 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:28:59.945923 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:00.143680 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:00.157009 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:00.160145 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:00.161513 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:00.169796 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:00.189260 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:00.223910 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:29:00.242084 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:29:00.261052 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:00.304405 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:00.376808 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:00.404932 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:00.408772 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:00.412024 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:00.416486 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:00.423096 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:00.445354 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:00.593111 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:00.649240 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:00.652981 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:00.653084 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:00.654124 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:00.670266 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:00.682588 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:00.699582 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:00.749526 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:00.834007 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:29:00.968600 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:01.099897 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:29:01.115854 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.125949 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:01.126163 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:01.138105 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:01.150038 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:01.173163 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:01.198453 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:01.395376 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:29:01.419584 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:01.437485 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.438388 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:01.441709 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:29:01.441879 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.444769 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:01.535758 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:01.536021 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:01.676685 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:01.681836 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.681857 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:01.685290 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:01.696023 948 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:01.696216 948 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
W20240502 08:29:01.700224 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:01.700368 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:29:01.703959 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.704394 948 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:01.707065 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:01.709275 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:01.709408 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:01.709908 948 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:01.757537 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:01.762351 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:01.853245 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:29:01.854194 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:29:01.955902 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.962064 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:01.962426 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:01.966078 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:01.972316 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:02.166698 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:02.182456 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:02.424808 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:02.436640 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:02.471521 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:02.510886 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:02.511960 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:02.544766 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:02.643724 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:02.657450 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:02.710305 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:02.711017 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:02.719560 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:02.729532 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:02.739665 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:02.825045 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:02.864907 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240502 08:29:02.865901 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:02.935485 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:02.937141 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:02.978408 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:02.979598 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:02.987555 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:02.996709 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:02.999840 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:03.070279 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:03.140612 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:03.152257 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:03.233253 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:03.247077 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:03.254055 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:29:03.263077 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:03.269081 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:03.269217 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:03.403234 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:03.404345 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:03.567476 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:03.604784 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:03.653136 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:03.677140 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:03.690115 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:03.693749 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:03.730962 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:03.740598 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:03.856155 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:29:03.899817 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:29:03.968077 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:03.977501 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:03.981802 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:03.982411 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:03.982698 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:03.983717 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:04.000115 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:04.015170 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:04.028604 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:04.138378 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:04.143520 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:04.206645 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:04.207118 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:04.207517 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:04.219449 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:04.223438 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:04.229179 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:04.447932 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:04.475575 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:04.475925 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:04.477630 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:04.477818 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:04.516518 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:04.632781 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:04.687340 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:04.693315 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:04.945243 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:04.951052 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:04.975864 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:04.987537 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:04.990041 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:05.022217 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:05.084424 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:05.094472 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:05.207233 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:05.224473 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:05.225792 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:05.226470 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:05.231319 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:05.257015 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:05.267155 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:05.307148 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:05.445742 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:05.461043 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:05.466584 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:05.468590 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:05.472294 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:05.481351 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:05.485432 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:05.505223 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:29:05.640455 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:05.698328 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:05.702325 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:05.713425 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:05.752359 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:05.775673 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 57226.2us
W20240502 08:29:05.777123 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:05.789727 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:05.877682 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:05.962687 1229 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:05.962946 1229 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:05.963757 1229 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:05.964682 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:05.965129 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:05.965245 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:05.965651 1229 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:06.088812 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:06.112155 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:06.218166 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:06.218938 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:06.220788 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:06.228168 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:06.236515 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:06.252101 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:06.330781 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:06.452860 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:29:06.465051 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:06.495223 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:29:06.499524 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:06.504215 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:06.507369 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:06.509440 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:06.520143 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:06.522627 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:06.552608 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:29:06.671865 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:06.697559 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:29:06.713341 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:06.733630 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:06.734599 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:06.735293 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:06.751564 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:06.865679 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:06.946918 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:29:07.007675 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:07.022286 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:07.023113 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:07.023205 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:07.073673 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:07.104413 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:07.213776 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:07.292496 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:07.425900 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:07.464087 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:07.466354 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:07.477631 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:07.491397 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:07.500385 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:07.591579 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:07.601425 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:07.690807 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:07.700181 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:07.716208 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:07.717563 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:07.742331 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:07.756680 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:07.793511 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:07.804456 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:07.815068 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:07.895328 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240502 08:29:07.961855 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:07.968868 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:07.976147 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:07.983944 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:07.984177 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:08.183450 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:29:08.192391 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:08.194154 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:08.201287 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:08.208359 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:08.220724 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:08.262912 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:08.300417 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:08.324363 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:08.477033 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:08.644327 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:08.727694 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:08.738543 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:08.739862 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:08.745532 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:08.767597 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:08.817797 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:08.824744 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:08.848182 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:29:08.904647 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:08.940479 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:08.963021 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:09.021065 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:09.038794 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:09.039464 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:09.039762 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:29:09.039935 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:09.040915 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:09.054595 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:09.056353 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:09.248595 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:09.266252 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:29:09.272130 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:09.285543 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:09.298992 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:09.299576 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:09.300530 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:09.314188 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:09.516745 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:09.547266 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:09.549515 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:09.550071 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:09.569584 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:09.576709 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:09.611102 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:09.633764 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:09.733457 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:29:09.935431 1229 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:09.935626 1229 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:09.936403 1229 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:09.941426 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:09.942649 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:29:09.942817 23406 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:09.947189 2276 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:09.974725 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:09.978960 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:09.987125 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:09.992307 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:09.997627 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:10.009950 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:10.106753 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:10.219206 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:29:10.227818 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:10.240293 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:10.244088 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:10.263692 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:10.279886 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:10.322412 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:10.352777 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:10.404791 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:10.507378 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:10.508466 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:10.517481 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:10.536191 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:10.537003 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:10.583930 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:10.674378 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:10.685478 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 08:29:10.741469 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:10.762495 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:10.767413 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:10.784817 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:10.792640 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:10.798667 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:10.854861 21880 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 74767.6us
W20240502 08:29:10.870795 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:10.955041 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:11.106709 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:11.160403 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:11.227039 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:11.232203 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:11.259940 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:11.259955 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:11.260123 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:11.320089 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:11.320503 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:11.394685 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:11.463904 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:11.484709 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:11.490746 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:11.498896 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:11.500102 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:11.505764 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:11.538424 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:29:11.568629 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:11.668483 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:11.690744 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:11.733407 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:11.746191 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:11.747421 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:11.747562 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:11.761992 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:11.973310 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:11.995476 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:29:12.030534 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:12.033248 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:12.035533 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:12.043099 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:12.048072 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:12.069679 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:12.100999 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:12.102078 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:12.325413 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:12.433290 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:12.457201 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:12.473488 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:12.490618 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:12.494446 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:12.530830 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:12.531399 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:12.733814 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:12.745337 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:12.747161 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:12.751353 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:12.763805 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:12.778118 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:12.846995 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:12.899356 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:12.900461 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:12.901037 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:12.912534 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:12.989123 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:12.995630 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:13.003129 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:13.006927 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:13.013692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:13.020977 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:13.024502 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:29:13.026000 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:29:13.029253 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:13.044730 1229 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:13.044942 1229 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:13.045656 1229 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:13.046975 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:13.082897 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:13.108922 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:13.109351 1229 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:29:13.239802 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:13.243439 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:13.249399 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:13.253052 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:13.257895 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:13.377566 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:13.388614 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:13.434738 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:13.634523 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:13.685895 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:13.688375 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:13.688498 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:13.696606 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:13.718358 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:13.719075 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:13.733373 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:13.939011 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:13.956650 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:13.963979 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:13.969620 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:13.973367 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:29:13.996563 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:14.005353 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:14.017120 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:14.023963 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:14.036972 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 63569.9us
W20240502 08:29:14.038592 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dc49c050280346db88d1323e30513c80: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:29:14.074272 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:29:14.084323 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:29:14.178613 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:14.224267 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:14.233424 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:14.252626 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:14.255755 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:14.275852 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:14.418100 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:29:14.419152 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:14.511626 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:14.519886 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:14.536978 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:14.548605 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:14.551777 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:14.604616 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:14.674878 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:14.758193 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:14.804826 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:14.862026 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:14.932190 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:14.954324 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:14.965096 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:14.967037 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:14.975107 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:14.989372 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:14.990468 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:15.165398 1229 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:15.165568 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:15.165588 1229 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:15.166280 1229 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:15.168021 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:15.173213 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:15.173429 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:15.173725 1229 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:29:15.196576 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:15.198953 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:15.203089 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:15.205530 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:15.228893 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:15.261360 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:15.411456 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:15.413583 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:15.418516 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:15.431548 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:15.437050 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:15.469370 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:15.614984 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:15.635895 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:15.637460 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:15.637972 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:15.638332 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:15.652694 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:15.660040 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:15.765300 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:15.870834 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:15.884977 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:16.023862 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 08:29:16.061432 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:16.085678 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:16.094831 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:16.097604 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:16.097877 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:16.107055 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:16.124838 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:16.151079 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:16.153065 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:16.319909 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:16.369381 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:16.385308 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:16.409341 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:16.431135 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:16.467471 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:16.489519 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:29:16.538009 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:16.610270 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:16.666545 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:16.669519 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:16.671352 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:16.685038 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:16.691041 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:16.715341 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:16.715507 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:16.847086 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:29:16.893805 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:16.913173 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:16.916282 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:16.930436 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:16.969280 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:16.983414 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:17.083158 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:17.180233 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:17.212448 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:17.222673 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:17.230461 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:17.416801 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:17.431560 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:17.444547 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:17.450978 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:17.461736 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:17.462630 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:17.489235 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:17.657847 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:17.665331 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:17.679728 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:17.690094 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:17.694854 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:17.700626 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:17.760972 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:17.833964 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:17.890918 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:17.914520 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:17.919283 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:17.935015 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:17.954406 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:17.976464 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:17.977452 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:18.027942 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:18.155635 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:18.167598 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:18.187777 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:18.196877 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:18.208299 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:18.218500 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:18.239689 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:18.267931 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:29:18.445710 1229 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:18.445919 1229 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:18.446698 1229 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:18.447697 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:18.448133 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:18.448266 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:18.448566 1229 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:18.488188 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:18.561702 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:18.576670 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:18.630065 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:18.631662 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:18.648823 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:18.655496 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:18.658830 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:18.688598 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:18.725735 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:18.920734 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:18.924144 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:18.943403 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:18.953642 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:18.970805 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:19.067876 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:19.095602 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:19.134924 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:29:19.165692 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:19.166908 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:19.184538 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:19.195264 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:19.196377 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:19.201962 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:19.261255 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:19.291661 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:19.359643 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:19.430025 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:19.443617 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:19.446368 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:19.456117 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:19.459590 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:19.518189 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:19.525003 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:19.568845 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:29:19.604252 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:19.737721 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:19.814287 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:19.815632 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:19.893093 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:19.901361 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:19.911121 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:19.934257 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:19.953636 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:19.961112 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:20.150508 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:20.169314 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:20.170169 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:20.170477 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:20.176992 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:20.193243 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:20.201193 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:20.394896 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:20.403908 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:20.407533 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:20.418164 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:20.421445 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:20.446432 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:20.467226 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:20.593837 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:29:20.620045 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:20.668336 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:20.685001 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:20.687487 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:20.689198 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:20.695227 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:20.706202 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:20.723845 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:20.802068 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:20.975087 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:21.091538 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:21.111452 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.125344 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:21.127175 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:21.128260 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.148249 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:21.189038 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:21.314554 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:21.332609 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:29:21.383457 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:21.383555 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.396169 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:21.405283 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.406351 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:21.475553 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:21.619534 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:21.620617 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:21.626992 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.627503 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:21.642688 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:21.708885 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:21.735564 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:21.795090 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:29:21.826289 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:21.836593 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:21.850229 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:21.893134 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:21.924175 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:21.939980 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:21.950364 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:21.958783 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:21.967531 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:22.162927 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:22.210008 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:22.281038 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:29:22.349325 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:22.367262 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:22.388659 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:22.389546 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:22.412187 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:22.443639 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:22.528745 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:22.548081 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:22.592900 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:22.634929 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:22.646646 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:22.657070 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:22.662479 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:22.666767 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:22.686151 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:22.891199 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:22.913240 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:22.913965 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:22.916014 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:22.942270 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:22.968961 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:22.976480 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:23.002223 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:23.003226 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:23.003840 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:23.151140 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:23.157328 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:23.158159 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:23.161093 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:23.162137 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:23.163532 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:23.312211 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:23.329584 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:23.370620 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:23.573901 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:23.575358 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:23.599967 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:23.611428 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:23.611943 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:23.671692 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:23.840703 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:23.865068 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:23.895071 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:23.895519 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:23.896138 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:23.904929 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:23.912402 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:24.010251 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:29:24.045913 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:24.118033 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:24.125844 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:24.130616 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:24.132552 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:24.140249 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:24.145000 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:24.324079 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:29:24.349642 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:24.361723 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:24.372112 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:24.385169 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:24.425956 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:24.431491 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:24.450429 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:24.455734 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:24.535867 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:24.619932 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:24.642561 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:24.706701 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:24.733994 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:24.878355 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:24.917374 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:24.925938 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:24.937700 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:24.953430 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:24.959209 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:25.100042 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:25.165570 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:25.176102 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:25.190250 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:25.201138 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:25.202634 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:25.207902 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:25.221410 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:25.223503 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:25.237751 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:25.259724 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:25.313154 3640 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:25.313376 3640 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:25.314179 3640 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:25.315228 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:25.315680 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:25.315794 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:25.316043 3640 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:29:25.432929 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:25.444547 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:25.461956 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:25.469067 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:25.482630 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:25.627018 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:25.627502 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:25.681094 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:25.689846 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:25.691735 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:25.693431 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:25.706830 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:25.722036 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:25.788460 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:25.796008 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:26.105978 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:26.118484 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:26.122918 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:26.130533 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:26.133303 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:26.137802 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:26.160573 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:26.175710 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:26.221375 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:26.353927 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:26.393815 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:29:26.398941 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:26.398975 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:26.434010 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:26.434798 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:26.450909 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:29:26.459997 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:26.605051 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:26.641827 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:26.664083 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:26.669072 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:26.686070 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:26.704941 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:26.857986 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:26.916847 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:29:26.927336 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:26.943553 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:26.950071 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:26.952811 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:26.954428 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:26.969736 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:27.052496 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:27.097472 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:27.106724 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:27.303122 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:27.321121 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:29:27.389890 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:27.420135 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:27.424983 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:27.443763 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:27.444396 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:27.455927 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:27.562306 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:27.565773 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:27.672972 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:27.679744 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:27.684259 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:27.686522 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:27.708483 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:27.719959 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:27.850152 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:27.878024 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:27.952888 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:27.968170 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:27.985853 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:28.002121 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:28.015880 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:28.016809 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:28.039073 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:29:28.077963 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:28.145022 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:28.209792 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:28.223130 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:28.225286 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:28.225911 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:28.254722 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:28.262979 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:28.288007 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:28.400096 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:28.438323 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:28.519061 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:28.544852 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:28.606909 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:28.649322 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:28.670169 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:28.673616 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:28.676602 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:28.682785 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:28.877048 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:28.912256 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:28.919407 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:28.931872 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:28.972671 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:28.986755 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:29.117213 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:29:29.139549 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:29.144141 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:29.162757 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:29.171268 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:29.188867 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:29.198410 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:29.213802 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:29.283092 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:29.334537 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:29:29.402251 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:29.475085 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:29.480720 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:29.497224 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:29.516254 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:29.533774 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:29.579342 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:29.626067 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:29:29.627066 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:29.808835 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:29.947777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:29.949175 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:29.954726 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:29.963546 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:29:29.973935 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:29.975045 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:29.976307 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:30.004290 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:30.009989 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.020241 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.031921 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:30.192363 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:30.195554 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:30.215582 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:30.222889 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:30.232503 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:30.399379 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:30.436026 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:30.457912 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:30.458766 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:30.469899 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:30.471834 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:30.487371 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:30.532507 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:30.597438 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:30.656997 4290 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:30.657232 4290 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:30.658025 4290 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:30.662487 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:30.668262 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:30.668483 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:30.669306 4290 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:30.670028 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:30.710075 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:30.719964 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:30.729197 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:30.737491 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:30.741032 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:30.749357 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.838541 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.887393 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.888331 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:30.999204 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:31.043799 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:31.149480 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:31.150210 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:31.163817 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:31.169193 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:31.176095 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:31.177685 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:31.429352 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:31.439481 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:31.471500 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:31.472458 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:31.472757 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:31.476379 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:31.483714 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:31.664153 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:31.685369 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:31.686254 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:31.688432 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:31.689038 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:31.698915 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:29:31.952896 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:31.954288 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:31.960340 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:31.961347 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:29:31.968851 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:31.971855 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:31.989428 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:29:31.996345 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:32.007597 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:32.029388 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:32.187032 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:32.282727 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:32.384989 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:32.385877 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:32.424894 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:32.425886 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:32.425879 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:32.444360 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:32.444967 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:32.450531 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:32.495060 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:32.518136 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:32.551266 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:29:32.704885 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:32.709614 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:32.729686 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:32.744619 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:32.774475 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:32.819458 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:32.854501 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:32.975404 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:32.990326 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:32.997879 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:33.013082 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:33.014892 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:33.020587 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:33.109833 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:29:33.114575 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:29:33.115511 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:33.134667 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:33.226424 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:33.232982 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:33.233395 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:33.236690 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:33.257393 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:33.302318 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:33.305537 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:33.316303 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:33.379398 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:33.533411 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:33.702365 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:33.703418 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:33.709748 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:33.721299 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:33.727363 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:33.810398 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:33.885716 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:33.969722 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:33.974408 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:33.994215 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:34.013020 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:34.025631 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:45373, user_credentials={real_user=slave}} blocked reactor thread for 63225.3us
W20240502 08:29:34.054054 21619 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 70672us
I20240502 08:29:34.056583 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:34.080641 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:34.138418 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:34.235836 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:34.250775 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:34.259554 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:34.270359 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:34.275936 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:34.404412 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:29:34.446820 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:34.488574 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:34.494482 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:34.504527 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:34.512913 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:34.513098 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:34.540714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:34.640857 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:34.823833 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:34.825846 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:34.837157 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:34.839313 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:34.861423 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:34.876226 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:34.888047 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:34.889725 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:34.971869 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:34.981942 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:34.988792 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:34.989542 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:35.006778 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:35.208151 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:35.209319 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:35.213539 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:35.218833 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:35.251765 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:35.276046 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:35.367789 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:35.378435 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:35.479436 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:35.489475 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:35.496371 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:35.507437 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:35.520788 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:35.535830 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:35.640841 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:35.658608 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:35.716343 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:35.726176 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:29:35.739418 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:35.750849 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:35.752436 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:35.754045 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:35.780370 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:35.882469 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:35.901878 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:35.960536 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:36.134457 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:36.146816 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:36.194984 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.212786 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:36.213929 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:36.227738 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.230973 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:36.292433 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:36.450573 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.469291 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:36.470405 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:36.470578 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:36.482640 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:36.559937 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:29:36.562619 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:36.675485 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:36.711783 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:36.714957 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:36.714999 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.716353 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:36.744984 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:36.950294 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:29:36.981105 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:36.992219 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.998468 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:36.999192 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:37.000461 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:37.087975 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:29:37.089735 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:37.145504 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:37.147472 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:37.249990 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:37.272975 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:37.448827 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:29:37.456027 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:37.486517 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:37.487509 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:37.489956 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:37.500356 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:29:37.513996 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:37.516041 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:37.534066 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:37.565595 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:37.643695 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:29:37.754976 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:37.782591 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:37.789233 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:37.800920 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:37.836467 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:37.842577 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:38.057265 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:38.074357 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:38.081843 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:38.088938 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:38.094437 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:38.108453 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:38.154206 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:38.205416 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:38.297493 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:38.306843 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:38.307133 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:38.310499 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:38.332762 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:38.347282 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:38.355281 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:38.385998 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:38.389825 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:38.479297 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:38.481374 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:38.625023 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:38.757673 4290 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:38.757866 4290 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:38.758778 4290 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:38.760442 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:38.761113 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:38.761245 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:38.761504 4290 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:29:38.780361 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:38.785678 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:38.792179 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:38.797060 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:38.826865 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:38.830341 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:38.836876 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:39.045644 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:39.071413 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:39.071484 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:39.072104 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:39.092304 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:39.134786 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:29:39.201087 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:39.288566 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:39.301749 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:39.306131 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:39.306596 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:39.318089 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:39.328307 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:39.525995 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:39.543759 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:39.549249 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:39.552912 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:39.556386 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:39.567490 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:29:39.589869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:39.765316 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:39.795337 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:29:39.840866 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:39.854285 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:39.862558 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:39.930990 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:39.937250 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:39.978911 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:40.000133 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:40.002801 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:40.010118 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:40.025195 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:40.028664 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:40.130159 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:40.193503 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:40.243412 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:40.244307 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:40.251303 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:40.263523 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:40.301148 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:40.394799 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:40.476857 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:40.491789 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:40.508867 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:40.519857 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:40.529526 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:40.603228 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:40.638270 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:40.725656 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:40.737032 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:40.760475 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:40.765189 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:40.766846 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:40.777287 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:40.779165 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:40.787053 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:40.791265 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:40.872406 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:40.908417 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:40.971181 4290 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:40.971418 4290 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:40.972111 4290 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:40.973081 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:29:40.973618 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:40.973714 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:40.973942 4290 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:40.982540 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:41.216555 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.216750 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:41.217309 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:41.239027 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:41.239856 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:41.283496 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:41.338949 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:41.448410 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.457271 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:41.477936 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:41.478986 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.488119 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:41.514730 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:41.586182 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:41.588356 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:29:41.671600 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.684123 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:41.684832 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.690237 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:41.694202 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:41.939939 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:41.958098 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:41.972273 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:41.977092 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:41.982571 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:29:41.996692 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:42.119273 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:42.132560 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:42.271845 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:42.329149 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:42.427841 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:42.436475 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:42.436861 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:29:42.440294 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:42.463378 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:42.463898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:42.471904 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:42.472733 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:42.505496 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:29:42.508083 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:42.563210 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:42.586422 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:29:42.619146 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:42.661895 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:42.675271 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:42.683418 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:42.689939 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:42.699007 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:42.901525 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:42.936146 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:42.939144 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:42.945134 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:42.961177 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:42.961547 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:43.023233 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:43.050673 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:43.098403 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:43.111912 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:43.204144 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:43.209062 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:43.211711 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:43.216997 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:43.231541 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:43.246423 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:43.251979 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:43.410426 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:43.471558 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:43.572796 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:43.630450 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:43.641952 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:43.647111 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:43.680315 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:43.680323 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:43.685444 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:43.710636 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:43.867666 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:43.896697 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:43.931788 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:43.935155 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:43.955992 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:43.969734 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:43.992928 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:44.112068 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:44.189180 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:44.212894 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:44.216106 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:44.229315 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:44.244258 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:44.280395 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:44.376807 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:44.439945 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:44.460106 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:44.460719 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:44.462810 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:44.463107 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:44.605988 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:44.683670 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:29:44.829216 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:44.905822 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:44.906477 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:44.918289 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:44.921474 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:44.922245 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:44.955906 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:44.964689 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:45.029456 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:29:45.095646 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:45.149037 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:45.152941 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:45.157558 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:45.163324 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:45.164345 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:45.168949 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:45.209558 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:45.214474 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:45.252207 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:45.400411 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:45.412315 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:45.416589 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:45.417199 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:45.428292 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:45.431708 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:45.526988 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:45.554786 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:45.555285 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:45.629137 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:45.650343 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:45.658526 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:45.662470 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:45.672345 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:45.682436 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:29:45.846545 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:29:45.847491 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:45.915066 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:46.018350 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:46.087532 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:46.096390 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:46.114436 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:46.117468 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:46.134169 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:46.146757 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:46.309645 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:46.335130 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:46.375679 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.404795 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:46.406528 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:46.410921 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:46.423400 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:46.428957 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:46.433236 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.599505 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.608038 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:46.624763 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.625449 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:46.628306 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:46.628476 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:46.709110 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:46.855862 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.876149 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:46.883697 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:46.901093 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:46.904795 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:46.905537 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:47.070669 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:47.076575 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:47.176694 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:29:47.210611 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:47.335038 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:47.341897 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:47.346478 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:47.356221 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:47.365885 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:47.450191 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:47.489415 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:47.514356 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:47.579342 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:47.585268 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:47.594626 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:29:47.598259 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:47.615613 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:47.618991 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:47.619591 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:47.705525 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:47.746816 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:47.848799 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:47.877738 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:47.883523 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:47.893214 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:29:47.901477 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:47.919997 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:47.926288 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:47.951706 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:48.025513 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:48.133358 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:48.141433 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:48.148648 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:48.160478 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:48.178622 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:48.229806 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:48.283787 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:48.297129 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:29:48.461684 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:48.465353 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:48.557638 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:48.592247 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:48.606405 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:48.607136 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:48.619901 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:48.622061 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:48.625362 4290 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:29:48.625594 4290 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:29:48.626317 4290 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:29:48.627703 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:48.629446 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
I20240502 08:29:48.629593 23406 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:29:48.630108 4290 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:29:48.651273 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:48.760377 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:48.800099 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:48.848348 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:29:48.859683 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:48.883472 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:48.886718 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:48.890283 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:48.891498 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:48.935966 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:48.950470 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:49.054924 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:49.090587 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:49.091444 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:49.115725 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:49.121969 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:49.158460 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:49.326126 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:29:49.339673 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:49.352669 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:49.356010 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:49.377930 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:29:49.378319 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:49.393465 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:49.464027 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:49.586654 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:49.823741 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:49.846561 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:49.868417 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:49.877169 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:49.915366 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:29:49.920755 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:49.942742 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:49.973553 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:49.974462 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:49.993746 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:50.156348 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 08:29:50.158783 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:50.179350 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:50.186978 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:50.187985 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:50.199478 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:50.201318 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:50.211966 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:50.225198 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:50.416996 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:50.432204 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:50.444943 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:50.463835 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:50.475461 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:50.500967 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 50276.6us
I20240502 08:29:50.506963 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:50.509774 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:50.531970 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:50.544417 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:29:50.673624 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:50.693830 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:50.697042 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:50.703352 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:50.713379 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:50.876543 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:29:50.877555 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:50.916054 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:51.140913 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:51.152194 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:51.161669 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:51.171186 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:51.172308 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:51.210471 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:51.226418 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:51.232949 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:51.359817 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:51.393393 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:51.397171 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:51.413332 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:51.415576 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:51.425477 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:51.432235 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:51.509723 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:51.583815 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:51.604602 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:51.605433 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:51.624217 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:51.624619 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:51.625638 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:51.631958 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:51.653575 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:51.896466 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:51.904943 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:51.910815 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:51.929483 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:51.949680 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:51.950727 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:52.059334 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:52.243921 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:52.316545 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:52.368561 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:52.386646 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:52.402693 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:52.412180 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:52.412770 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:52.427342 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:52.430246 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:52.484869 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:52.526329 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:52.598407 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:29:52.616195 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:52.616437 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:52.627696 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:52.641059 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:52.644080 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:52.647392 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:29:52.735378 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:52.821231 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:29:52.843801 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:52.850437 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:29:52.867048 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:52.869014 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:52.890873 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:52.904431 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:52.908922 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:29:52.996052 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:53.010912 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:53.084884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:53.109170 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:53.109874 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:53.128196 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:53.139950 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:53.143577 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:53.146808 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:53.363994 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 08:29:53.534341 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:53.538758 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:53.543634 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:53.552011 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:53.561054 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:53.561708 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:53.564127 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:53.825219 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:53.830152 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:53.842108 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:29:53.865381 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:53.893566 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:53.893777 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:53.917109 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:53.934684 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:53.965399 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:54.044999 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:54.105295 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:54.119659 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:54.134647 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:54.136227 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:54.156574 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:54.157753 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:54.161478 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:54.245954 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:29:54.357733 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:54.378720 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:54.386531 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:54.390156 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:54.399574 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:54.564360 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:54.639006 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:54.737305 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:29:54.756158 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:54.804226 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:54.805550 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:54.846585 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:54.856655 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:54.873989 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:54.881314 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:54.941480 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:54.975833 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:55.039333 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:55.105934 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:55.129343 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:55.139364 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:55.143630 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:55.147910 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:55.148705 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:55.185045 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:29:55.346179 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:55.353948 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:55.355492 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:29:55.360283 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:55.371860 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:55.398277 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:55.458915 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:55.526654 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:55.539098 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:29:55.542472 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:55.603817 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:55.623594 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:55.626000 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:55.628821 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:55.650978 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:55.788051 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:55.806537 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:29:56.035634 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.054152 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:56.059108 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.060771 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:56.061921 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:56.092541 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:56.144145 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:56.237943 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:56.249441 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:29:56.289664 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:56.290112 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.293223 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:56.302263 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:56.316668 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.316833 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:56.382591 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:56.390486 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:29:56.512238 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:56.546305 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.552095 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:56.552451 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:56.572079 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:56.574786 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:56.595288 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:56.686475 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:56.765030 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:56.780808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:56.785419 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:56.789999 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:56.817852 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:56.928620 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:29:57.107939 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:57.231756 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:57.253458 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:57.256527 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:57.273957 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:57.284142 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:57.287654 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:29:57.346920 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:57.347213 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:57.373112 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:57.396840 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:57.457160 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:57.469067 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:29:57.496506 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:57.502319 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:57.516927 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:57.530201 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:57.531222 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:57.556157 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:29:57.708798 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:57.710106 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:57.710232 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:57.725904 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:57.749403 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:57.803393 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:29:57.815829 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:57.975018 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:57.980610 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:57.983232 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:29:57.984074 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:57.985649 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:57.995577 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:58.000713 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:58.013139 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:58.043897 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:29:58.065815 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:29:58.116881 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:29:58.256140 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:58.423446 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:58.429327 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:58.433836 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:58.442579 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:29:58.454316 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:58.454733 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:58.477263 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:58.557796 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:29:58.646446 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:58.664867 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:58.666661 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:58.672109 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:58.672796 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:58.703480 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:29:58.709411 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:29:58.747164 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:58.937292 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:58.938104 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:29:58.943048 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:58.959986 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:58.960363 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:58.975895 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:59.010411 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:59.060489 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:59.122710 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:29:59.183305 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:59.193543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:59.195161 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:59.202044 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:59.225171 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:59.448185 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:59.494267 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:29:59.620896 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:59.624722 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:59.645511 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:29:59.654194 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:29:59.660550 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:29:59.743000 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:29:59.750386 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:29:59.905237 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:59.937815 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:29:59.938882 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:29:59.952867 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:29:59.959451 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:29:59.962824 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:29:59.969564 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:29:59.983738 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:29:59.997202 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:30:00.117895 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:30:00.158746 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:00.161603 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:00.174825 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:00.180955 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:00.184562 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:00.199481 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:30:00.243832 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:00.350430 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:00.412529 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:00.432055 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:00.434607 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:00.437283 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:00.448235 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:00.453393 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:00.466840 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:00.571075 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:30:00.573468 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:30:00.595880 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:00.755648 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:30:00.882416 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:00.903419 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:00.920342 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:00.921818 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:00.931561 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:01.032789 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:01.093652 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:01.148495 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:01.174868 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:01.190747 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:01.208206 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:01.215150 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:01.242517 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:01.242929 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 74442.4us
W20240502 08:30:01.252257 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:30:01.276136 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:01.308581 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:01.411284 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:01.446461 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:30:01.455827 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:01.463374 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:01.480650 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:01.483219 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:01.494732 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:01.494833 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:01.569418 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:30:01.652937 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:01.718122 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:01.725781 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:01.728848 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:01.732017 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:01.754415 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.181813 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.198154 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:02.198348 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:02.216403 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.218027 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:02.271996 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:02.409617 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:02.446086 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.453836 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:02.457342 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:02.470008 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.470063 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:02.474565 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:02.546566 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:02.619685 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:02.659467 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:02.666669 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:02.669085 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:02.675418 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:02.678588 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:02.681762 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:02.698693 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:02.716898 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:02.801975 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:02.839547 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:02.840814 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:02.866477 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:02.922266 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:02.926165 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:30:02.928081 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:30:02.930976 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:02.981432 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:02.982421 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:02.993620 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:03.054076 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:03.057457 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:03.147300 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:30:03.219285 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:30:03.422614 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:03.429445 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:03.438781 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:03.444190 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:03.462826 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:30:03.464689 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:03.466359 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:03.476145 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:03.477047 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:03.477174 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:03.680927 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:03.684968 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:03.698637 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:03.699584 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:03.736718 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:03.741226 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:03.829188 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:30:03.989308 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:03.998786 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:03.998942 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:03.999660 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:04.005071 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:04.029358 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:04.049583 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:04.134299 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:04.175779 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:04.241008 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:04.242216 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:04.252499 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:04.268064 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:04.272219 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:04.519059 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:04.720566 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:04.723368 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:04.726783 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:04.743328 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:04.753911 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:04.966993 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:04.983788 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:04.986088 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:04.997400 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:05.001082 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:05.007706 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:05.014606 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:05.024976 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:05.055893 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:05.111742 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:05.220643 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:05.221617 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:05.225987 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:05.235579 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:05.244741 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:05.254841 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:05.280391 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:30:05.289587 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:05.304580 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:05.314831 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:05.332008 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:05.421391 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:05.459149 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:05.467272 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:05.480994 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:05.483927 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:05.485153 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:05.491406 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:05.582693 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:05.707828 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:05.805980 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:30:05.822281 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:05.908720 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:05.923087 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:05.928362 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:05.954334 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:05.965036 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:05.989725 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:06.011502 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:06.039794 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:06.070079 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:06.149847 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:06.156949 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:06.172099 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:06.174379 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:06.195977 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:06.307484 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:06.377660 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:06.393111 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:06.402014 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:06.405973 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:06.406436 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:06.414068 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:06.526769 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:30:06.537164 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:06.629650 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:30:06.692214 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:06.697304 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:06.700431 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:06.704180 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:06.716746 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:06.732132 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:06.895789 9511 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:06.896027 9511 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:06.896914 9511 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:06.897933 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:06.898300 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:06.898437 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:06.898715 9511 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:06.929587 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:07.145447 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:07.151995 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:07.154181 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:07.163553 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:07.193326 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:07.277019 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:07.292572 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:30:07.409440 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:07.413933 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:07.430397 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:07.434018 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:07.436178 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:07.490770 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:07.536739 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:07.635607 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:07.656827 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:07.674216 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:07.675546 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:07.678107 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:07.713562 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:07.770287 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:07.836131 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:07.873383 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:30:07.921963 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:07.948863 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:07.960067 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:07.964320 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:07.986743 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:30:07.997898 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:08.013572 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:08.021538 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:08.082504 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:08.088560 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:08.129503 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:08.204569 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:30:08.207067 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:08.285189 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:08.407622 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:08.418118 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:08.423348 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:08.423755 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:30:08.424943 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:08.444772 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:08.481062 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:30:08.483124 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:08.533800 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:08.559837 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:08.648330 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:08.648742 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:08.651085 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:08.662375 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:08.667585 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:08.668087 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:08.791705 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:30:08.907966 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:08.923911 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:08.940631 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:08.949918 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:30:08.958173 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:08.964201 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:09.018370 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:09.152877 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:09.171978 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:09.173089 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:09.175436 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:09.175768 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:09.220845 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:09.342995 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:09.555924 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:09.633175 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:09.656224 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:09.664135 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:09.675469 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:09.691174 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:09.694530 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:09.962903 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:09.983623 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:09.986552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:09.986982 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:09.999398 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:10.014640 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:10.041898 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:10.086267 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:10.140388 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:10.174254 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:10.194658 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:10.195641 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:10.212586 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:10.231606 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:10.233433 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:10.343753 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 08:30:10.414952 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:10.423466 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:10.461786 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:10.468549 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:10.479624 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:10.482257 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:10.486176 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:10.496982 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:10.527154 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:10.551683 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:10.578261 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:10.600255 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:10.607676 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:10.681994 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:10.734803 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:30:10.925810 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:10.939291 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:10.950009 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:10.960353 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:10.988749 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:11.028309 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:11.038898 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:11.138124 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:11.139065 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:11.204795 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:11.205196 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:11.220137 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:11.229586 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:11.229897 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:11.345320 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:11.432724 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:11.435941 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:11.438055 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:11.442040 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:11.445434 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:11.548274 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:11.567447 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:11.659313 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:11.698310 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:11.704413 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:11.709581 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:11.716905 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:11.728834 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:12.003583 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:12.095948 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:12.108080 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:12.153481 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:12.155586 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:12.156579 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:12.156571 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:12.192370 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:12.343467 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:12.413254 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:12.414340 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:12.431536 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:12.445109 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:12.465575 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:12.528916 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:12.531107 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:12.566478 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:12.676400 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:12.680025 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:12.691269 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:12.692348 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:12.698103 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:12.698143 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:12.714669 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:12.848520 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:12.929445 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:12.956702 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:12.960242 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:12.977710 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:12.984228 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:12.987501 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:12.987643 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:13.000906 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:13.005226 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:13.045918 21397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 08:30:13.047225 21397 catalog_manager.cc:4573] TS fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111)
W20240502 08:30:13.047922 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:30:13.123152 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:30:13.137542 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:13.144587 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:13.238662 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:30:13.281536 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:13.354255 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:13.423749 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:13.443025 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:13.445392 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:13.460206 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:13.469657 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:13.551544 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:30:13.646605 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:13.672358 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:13.673233 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:13.677776 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:13.686645 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:13.697553 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:13.703888 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:30:13.759423 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:13.767045 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:13.922365 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:13.932410 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:13.953672 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:13.966893 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:13.977173 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:14.101361 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:14.149807 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
W20240502 08:30:14.153362 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:14.192894 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:14.196478 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:14.206101 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:14.211108 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:14.223626 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:14.225360 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:14.246999 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:14.559379 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:14.663305 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:14.671422 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:14.676663 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:14.679754 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:14.683547 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:14.692021 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:14.820127 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:14.947934 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:14.959448 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:14.973589 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:14.979821 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:14.980677 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:14.980826 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:15.128885 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:15.197014 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:15.197710 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:15.200309 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:15.203228 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:15.210669 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:15.212311 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:15.272442 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:15.291618 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:15.345594 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:15.379981 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:15.421415 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:30:15.423134 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:15.450142 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:15.451290 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:15.458710 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:15.479805 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:15.487543 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:15.500650 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:15.540037 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:15.653865 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:30:15.704790 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:15.740514 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:15.825510 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:15.895457 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:15.936050 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:15.946758 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:15.965097 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:15.966737 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:15.998963 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:15.999799 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:16.135952 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:16.217211 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:16.221024 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:16.224045 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:16.234213 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:16.240660 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:16.279081 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:16.407538 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:16.444844 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:16.445434 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:16.445917 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:30:16.446004 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:16.447379 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:16.460781 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:16.483652 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:16.517639 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:16.678097 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:16.684509 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:16.699189 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:16.700285 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:16.711117 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:16.826133 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:17.107062 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:17.108947 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:17.123440 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:17.124397 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:17.132686 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:17.191848 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:17.209820 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:17.276230 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:17.342759 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:30:17.437981 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:17.443921 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:17.447161 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:17.455461 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:17.479990 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:17.535820 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:17.604892 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:17.682655 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:17.699221 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:17.702638 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:17.707966 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:17.718433 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:17.725945 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:17.801625 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:17.824869 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:17.965271 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:17.982731 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:17.991555 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:17.998399 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:18.004546 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:18.004693 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:18.019881 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:18.023651 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:30:18.029085 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:18.074401 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:18.192180 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:18.237275 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:18.310559 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:18.432740 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:30:18.444365 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:18.448869 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:18.456743 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:18.458039 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:18.461342 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:18.472625 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:18.525894 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:18.583161 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:30:18.627036 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:18.665335 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:18.689271 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:18.692626 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:18.713982 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:18.716368 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:18.724294 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:18.730878 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:30:18.841413 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:18.910785 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:18.951789 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:18.961243 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:18.961500 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:18.963042 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:18.967646 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:18.998343 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:30:18.999925 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:19.019560 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:19.200654 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:19.211763 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:19.214747 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:19.216441 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:19.222527 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:19.236716 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:19.258244 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:19.625392 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:19.648854 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:19.659281 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:19.664251 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:19.665096 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:19.728662 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:19.872045 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:19.872274 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:19.943558 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:19.949322 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:19.968257 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:19.987643 23407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:46845, user_credentials={real_user=slave}} blocked reactor thread for 54163.6us
W20240502 08:30:20.005331 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:20.008122 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:20.018533 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:20.192142 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:20.198774 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:20.207427 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:20.208863 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:20.214458 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:20.237501 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:20.305922 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:20.382704 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:20.427935 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:20.436868 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:20.437330 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:20.454957 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:20.455390 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:20.513187 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:20.516727 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:30:20.571305 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:30:20.572446 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:20.614606 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:20.686229 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:20.807411 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:20.807969 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:20.863322 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:20.892042 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:20.905330 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:20.928382 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:20.933737 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:20.939963 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:21.017531 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:21.043609 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:21.098455 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:21.130507 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:21.134124 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:21.134595 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:21.150381 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:21.164592 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:21.251791 11434 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:21.252023 11434 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:21.252822 11434 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:21.253944 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:21.254369 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:21.254462 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:21.254709 11434 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:21.331291 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:21.364749 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:21.371371 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:21.386341 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:21.395064 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:21.397468 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:21.405400 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:21.407131 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:21.431342 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 67212.9us
W20240502 08:30:21.480958 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:21.615420 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:21.626919 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:21.627583 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:21.636309 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:21.639739 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:21.658488 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:21.692056 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:21.726833 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:30:21.867978 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:22.072011 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:22.074868 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:22.075973 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:22.089924 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:22.103857 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:22.306663 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:22.315138 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:22.319763 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:22.323750 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:22.339232 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:22.371572 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 08:30:22.406507 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:22.557555 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:22.572187 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:22.579914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:22.581032 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:22.599363 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:22.603986 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:22.631691 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:22.707602 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:22.713654 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:22.769896 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:22.777251 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:22.847885 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:22.853240 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:22.870575 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:22.877135 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:22.880905 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:23.019345 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:23.024464 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:23.073379 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:23.226178 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:23.250603 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:23.278893 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:30:23.322304 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:23.324745 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:23.326611 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:23.341714 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:23.364681 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:23.395170 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:23.470127 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:23.515159 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:23.515499 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:30:23.570164 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:23.584054 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:23.584638 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:23.602501 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:23.608162 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:23.623308 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:23.692333 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:23.758572 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:30:23.760542 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:23.787667 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:23.804108 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:23.814391 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:23.845984 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:23.854694 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:23.903564 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:24.052832 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:24.064054 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:24.078356 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:24.092936 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:24.095563 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:24.119046 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:24.133036 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:24.173760 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:24.174549 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:30:24.340422 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:24.352290 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:30:24.515172 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:24.533030 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:24.535533 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:24.538372 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:24.551054 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:24.559689 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:24.766729 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:24.775295 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:24.791450 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:24.793906 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:24.801003 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:24.997452 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:25.050740 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:25.062103 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:25.064373 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:25.066787 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:25.073771 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:25.100432 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:25.228816 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:25.319445 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:25.319780 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:25.338351 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:25.352293 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:25.352559 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:25.361038 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:25.366686 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:25.375306 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:25.383888 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:25.454844 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:25.514678 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:25.544050 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:25.662588 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:30:25.692770 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:25.774147 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:30:25.806725 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:25.809850 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:25.821508 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:25.839990 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:25.898958 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:30:25.938490 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:25.939605 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:25.966794 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:25.979182 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:26.036681 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:26.119542 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:26.119958 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:26.127116 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:26.140962 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:26.150682 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:26.189514 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:26.293080 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:26.332558 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:26.333963 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:26.340087 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:26.353477 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:26.362516 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:26.368611 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:26.550729 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:26.602169 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:26.613617 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:26.618036 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:26.619824 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:26.639035 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:26.765098 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:26.787262 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:26.824501 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:26.909776 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:26.945397 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:27.048628 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:27.050016 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:27.052649 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.075158 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.088953 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:27.303591 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.317492 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:27.332345 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:27.339041 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:27.370764 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:27.438596 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:27.526870 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:27.616950 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.634955 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:27.641780 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:27.650897 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.657760 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:27.668381 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:27.871340 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:27.883917 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:27.893103 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:27.914746 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:27.921154 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:27.921602 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:27.928391 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:27.950150 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:27.951401 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:27.967808 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:27.981381 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:28.041744 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:28.057600 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:28.121647 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:30:28.396486 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:28.403507 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:28.422394 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:28.433713 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:30:28.439673 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:28.440526 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:28.440871 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:28.494657 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:28.525744 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:28.633272 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:28.633663 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:28.657956 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:28.662696 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:28.661367 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:28.674469 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:28.689628 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:30:28.692214 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:28.716624 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:28.733393 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:28.813349 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:30:28.818764 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:28.888876 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:28.905406 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:28.909828 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:28.951117 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:28.958355 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:28.971532 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:29.168134 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:29.176981 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:29.182786 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:29.184155 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:29.185241 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:29.188005 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:29.322564 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:29.423933 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:29.444715 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:29.474967 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:29.562893 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:30:29.598513 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:29.613579 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:29.614809 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:29.618335 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:29.641669 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:29.832409 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:29.833477 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:29.897346 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:29.906795 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:29.918448 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:29.920970 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:29.948137 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:30.116787 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:30.125769 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:30.140362 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:30.149729 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:30.155354 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:30.159538 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:30.169692 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:30.250411 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:30.263803 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:30.316745 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:30.334357 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:30.393965 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:30.409875 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:30.410215 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:30.416316 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:30.417747 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:30.434945 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:30.440817 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:30.477306 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:30.751730 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:30:30.817268 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:30:30.880460 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:30.883751 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:30.920054 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:30.939918 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:30.941696 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:30.959013 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:30.985638 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:30.997653 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:31.000061 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:31.005911 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:31.170812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:31.178179 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:31.179343 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:31.208446 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:31.230448 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:31.262356 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:31.265714 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:31.426587 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:30:31.435914 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:31.451261 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:31.452153 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:31.458878 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:31.478819 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:31.666601 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:31.676215 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:31.694619 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:31.697415 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:31.697906 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:31.703352 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:31.720603 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:31.724471 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:31.915216 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:31.997763 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:32.136920 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:32.145159 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:32.155190 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:32.162349 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:32.185545 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:32.186898 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:32.188486 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:32.221707 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:32.374279 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:32.387511 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:32.419802 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:32.448676 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:32.454420 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:32.473810 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:32.567036 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:32.592900 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:32.629853 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:32.641872 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:32.647537 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:32.659421 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:32.667500 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:32.676870 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:32.785429 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:32.843322 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:32.903999 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:32.905727 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:32.933943 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:32.945366 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:32.952292 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:32.953368 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:32.975781 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:32.987375 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:33.031035 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:33.090030 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:30:33.154476 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:33.245555 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:33.408742 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:33.417392 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:33.425638 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:33.449550 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:33.466598 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:33.474170 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:33.484205 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:33.592545 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:33.600564 12441 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:33.600811 12441 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:33.601559 12441 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:33.602726 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:33.602864 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:33.602941 23406 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:33.603365 12441 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:33.648437 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:33.681236 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:33.683149 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:33.688057 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:33.696774 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:33.698076 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:33.710987 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:30:33.714066 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:33.848647 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:30:33.851843 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:33.935451 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:33.953418 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:33.968537 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:33.973186 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:33.977643 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:34.043148 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:34.212402 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.218022 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:34.223915 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.225529 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:34.226382 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:34.328702 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:34.353889 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:34.450773 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:30:34.515981 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:34.533056 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:34.566079 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:34.614907 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:34.628881 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:34.641554 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:34.645381 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.667876 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.671973 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:34.893164 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.911907 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:34.915115 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:34.936100 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:34.937345 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:34.967010 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:30:34.971566 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:35.021706 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:35.045697 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:35.119642 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:35.122747 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:35.128690 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:35.133455 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:35.135623 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:35.143676 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:35.162328 12441 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:35.162509 12441 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:35.163142 12441 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:35.164762 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:35.165350 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:35.165454 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:35.165727 12441 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:35.180375 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:35.285063 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:35.354197 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:35.354559 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:35.358702 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:35.384992 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:35.411528 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:35.412197 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:35.545076 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:30:35.555828 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:30:35.695775 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:35.745385 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:35.776472 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:35.779053 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:30:35.780076 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:35.850812 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:35.882669 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:35.892311 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:35.926409 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:35.929081 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:35.980813 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:36.009387 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:36.081897 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:36.125377 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:36.135192 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:36.139219 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:36.139233 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:36.159489 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:36.220526 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:36.341172 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:36.341967 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:36.342922 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:36.348815 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:36.363680 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:36.500630 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:36.513041 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:36.591419 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:36.612596 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:36.618057 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:36.631672 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:36.639106 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:36.791726 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:36.883898 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:36.908385 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:36.990522 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:37.033444 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:37.050319 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:37.052461 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:37.057380 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:37.107403 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:37.122031 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:37.128643 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:37.138144 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:37.227465 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:37.316577 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:37.330227 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:37.337473 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:37.341477 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:37.342059 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:37.347334 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:37.372491 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:37.496945 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:37.557348 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:37.563306 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:37.566752 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:37.572405 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:37.581601 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:37.598553 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:37.737090 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:37.779011 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:37.799978 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:37.804358 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:37.841156 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:37.863023 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:37.918057 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:37.953805 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:37.985455 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:38.192162 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:38.229761 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:38.230480 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:38.254498 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:38.278847 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:30:38.380928 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:38.409052 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:38.421278 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:38.422140 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:38.431550 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:38.439435 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:38.492614 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:38.565847 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:30:38.618395 13654 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:38.618628 13654 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:38.619489 13654 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:38.620855 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:38.621083 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:38.621219 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:38.621667 13654 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:38.626520 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:38.644491 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:38.659716 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:38.660482 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:38.683817 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:38.704020 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:38.734642 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:30:38.873390 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:30:38.899571 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:38.902618 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:38.910751 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:38.921309 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:38.930944 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:38.939832 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:38.970686 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:39.163151 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:39.164899 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:39.183387 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:39.185909 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:39.186740 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:39.276330 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:39.314250 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:39.386057 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:39.406875 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:39.502075 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:39.532676 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:39.543517 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:39.552995 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:39.589828 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:39.602000 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:39.602777 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:39.606684 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:39.607424 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:39.625977 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:39.765137 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:39.773815 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:39.882294 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:39.889636 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:39.890959 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:39.907931 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:39.909662 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:39.933682 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:40.136854 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:40.138423 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:40.146652 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:40.151875 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:40.161227 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:40.184366 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:40.189702 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:30:40.233987 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:40.242344 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:40.366128 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:40.375101 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:40.393723 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:40.416009 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:40.435007 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:40.435351 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:40.464323 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:40.650046 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:40.653816 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:40.724340 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:40.725247 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:40.783463 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:30:40.887701 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:40.890753 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:40.902288 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:40.923602 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:30:40.929631 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:40.930758 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:41.061425 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:41.132373 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:41.156585 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:41.159092 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:41.166144 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:41.185724 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:41.194659 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:41.196627 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:41.382656 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:41.401353 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:41.407438 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:41.437103 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:41.455085 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:41.611451 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:41.642858 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:41.645458 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:41.660882 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:41.666273 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:41.686911 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:41.705334 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:41.788654 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:41.825237 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:41.870998 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:41.931838 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:41.931979 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:42.004979 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:42.037142 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:42.086756 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:42.087059 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:42.111003 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:42.114027 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:42.129839 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:42.133261 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:42.137144 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:42.221246 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:30:42.362744 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:42.370484 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:42.399662 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:42.404369 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:42.404865 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:42.405335 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:42.604597 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:42.617787 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:42.620206 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:42.626665 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:42.629022 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:42.642848 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:42.652977 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:42.659698 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:42.707067 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:42.887081 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:42.902344 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:42.922531 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:42.940760 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:42.956552 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:42.973624 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:43.095727 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:43.128715 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:43.182745 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:30:43.185798 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:43.279182 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:30:43.358800 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:43.359418 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:43.360385 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:43.392343 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:43.415122 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:43.446477 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:43.540550 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:43.607874 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:43.609361 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:43.610198 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:43.625908 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:43.636137 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:43.710059 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:43.813263 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:43.886605 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:43.915997 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 08:30:43.917650 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:30:43.920354 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:43.920713 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:43.926017 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:43.926652 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:43.965097 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:44.159129 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:44.166028 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:44.166155 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:44.179641 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:44.182992 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:44.258008 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:44.285437 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:44.305189 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:44.321362 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:44.333902 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:44.451823 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:44.534281 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:44.625155 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:44.630371 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:44.634661 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:44.638801 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:44.654443 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:44.656878 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:44.660014 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:44.757714 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:44.766353 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:44.861218 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:44.892050 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:44.902415 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:30:44.906127 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:44.907913 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:44.914955 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:44.915901 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:45.113685 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:45.158960 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:45.178720 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:45.180251 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:45.181195 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:45.185910 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:45.265206 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:45.291682 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:45.344841 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:45.362504 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:45.393446 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:45.411723 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:45.418292 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:45.428248 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:45.431064 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:30:45.446626 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:45.510838 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:30:45.705962 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:45.862696 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:45.877813 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:30:45.879364 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:45.879916 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:30:45.882606 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:45.887746 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:45.908605 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:45.934110 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:46.013418 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:46.112047 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:46.149645 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:46.162303 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:46.172281 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:46.182824 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:46.208367 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:46.394577 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:46.434816 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:46.440721 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:46.457113 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:46.469704 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:46.532073 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:46.588326 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:46.696602 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:46.706473 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:46.710198 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:46.718730 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:46.724759 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:46.751992 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:46.759696 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:46.801548 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:46.863691 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:46.918236 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:46.957718 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:47.012617 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:47.017069 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:47.195186 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:47.199071 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:47.204792 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:47.211220 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:47.225872 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:47.238222 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:47.239708 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:47.345403 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:47.356711 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:47.464058 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:47.465672 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:47.475107 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:47.483220 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:47.494225 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:47.536883 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:47.560346 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:47.568324 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:47.671602 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:47.673336 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:47.687335 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:47.689549 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:47.691475 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:47.713059 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:30:47.788718 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:47.934388 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:47.935612 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:47.936045 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:47.967713 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:47.987419 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:48.001482 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:48.165503 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:48.259837 13947 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:30:48.260079 13947 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:30:48.261122 13947 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:30:48.261806 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:30:48.263782 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:48.263911 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:30:48.264262 13947 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:30:48.273654 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:48.287025 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:30:48.338164 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:30:48.384527 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:48.413985 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:48.425570 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:48.434070 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:48.443393 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:48.448258 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:48.665061 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:48.669951 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:48.685111 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:48.686923 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:48.694490 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:48.695703 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:48.739192 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:48.920089 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:30:48.928066 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:30:48.935127 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:48.939019 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:48.956142 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:48.978427 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:48.980055 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:48.980716 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:49.053316 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:49.119740 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:49.158358 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:49.174153 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:49.211585 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:49.214828 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:49.224735 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:49.234282 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:49.259724 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:49.322575 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:49.366806 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:49.410935 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:49.459009 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:49.525171 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:49.641662 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:49.642768 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:49.661015 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:49.668524 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:49.668859 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:49.672940 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:49.899542 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:49.908465 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:49.912992 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:49.914796 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:49.950564 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:49.965940 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:49.968632 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:49.980695 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:50.017477 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:50.018275 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:50.071728 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:30:50.130499 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:50.162056 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:50.172796 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:50.175361 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:50.175630 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:50.177498 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:50.396035 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:50.396884 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:50.408020 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:50.416213 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:50.440222 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:50.442814 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:50.450830 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:50.572625 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:50.736430 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:50.852820 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:50.905848 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:30:50.907254 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:50.910872 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:50.917459 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:50.934355 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:50.938417 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:50.948793 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:50.970108 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:51.147805 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:51.160185 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:51.161984 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:51.175927 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:51.178532 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:51.215051 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:51.344412 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:51.394845 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:51.411669 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:51.426621 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:51.457213 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:51.463644 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:51.464071 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:51.468956 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:51.472178 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:51.552372 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:51.676409 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:51.686210 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:51.691759 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:51.694533 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:51.694721 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:51.721118 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:51.777467 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:51.904598 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:51.934145 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:51.998004 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:52.051319 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:52.053645 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:52.113440 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:52.113992 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:52.119340 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:52.126600 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:52.150774 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:52.292289 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:52.334766 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:52.335471 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:52.343526 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:52.352896 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:52.378808 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:52.383843 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:52.422022 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:30:52.513901 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:30:52.580319 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:52.591928 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:52.600786 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:52.630343 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:52.638406 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:52.657650 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 70856.4us
I20240502 08:30:52.661357 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:52.692447 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:52.790794 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:52.823596 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:52.909049 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:52.917255 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:52.930822 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:52.931290 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:52.955092 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:53.068946 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:53.148170 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:53.261986 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:53.356689 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:53.373255 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:53.377931 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:53.391685 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:53.396663 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:53.415292 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:53.448542 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:30:53.593832 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:30:53.643049 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:53.645838 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:53.654824 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:53.660624 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:53.661379 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:53.820814 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:53.877360 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:53.908973 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:53.930888 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:53.934957 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:53.939371 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:53.943270 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:30:53.944705 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:53.951004 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:53.965514 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:54.007972 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:54.023708 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:54.094516 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:30:54.196887 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:54.201205 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:54.204355 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:54.214006 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:54.215540 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:54.318682 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:54.354173 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:54.376269 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:54.380419 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:54.391146 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:54.492063 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:54.551321 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:54.621073 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:54.621541 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:54.623371 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:54.636871 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:54.656154 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:30:54.656826 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:54.708899 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:30:54.730787 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:54.890903 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:54.897547 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:54.897989 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:54.915486 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:54.934588 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:55.018405 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:30:55.142413 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:55.146345 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:55.154263 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:55.160020 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:55.172101 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:55.174845 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:55.177860 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:55.223650 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:55.247217 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:55.366683 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:55.377650 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:55.385881 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:55.393487 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:55.410805 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:55.429042 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:55.474193 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:55.563117 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:55.636881 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:55.822737 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:30:55.840662 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:30:55.911178 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:55.935519 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:55.936678 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:55.937161 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:55.945722 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:30:55.984040 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:30:56.021384 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:56.154552 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:56.164911 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:56.169219 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:56.182869 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:56.194634 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:56.203034 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:30:56.405043 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:56.414615 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:56.432531 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:56.432854 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:56.447763 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:56.505559 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:56.532236 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:56.545964 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:56.596737 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:56.614885 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:56.663894 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:56.672016 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:56.695355 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:56.713093 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:56.725701 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:56.789654 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:56.797565 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:56.798257 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:56.807411 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:56.846727 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:56.929018 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:56.953016 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:56.963320 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:30:57.124411 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:57.155707 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:57.159423 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:57.161041 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:57.171797 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:30:57.213662 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:30:57.336663 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:57.367411 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:57.380900 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:57.385408 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:57.388820 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:57.430780 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:57.483981 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:30:57.628676 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:57.630424 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:57.630981 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:57.639737 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:57.647084 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:57.703891 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:57.739320 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:57.742758 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:30:57.895812 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:57.897245 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:57.897404 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:57.906610 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:57.919711 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:57.934671 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:57.950994 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:58.176111 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:30:58.197647 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:58.268620 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:30:58.328881 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:58.333324 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:58.340910 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:58.342402 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:58.343744 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:58.350499 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:58.601769 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:58.614288 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:58.618639 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:58.621874 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:58.636278 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:58.800567 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:58.838759 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:58.859079 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:58.859716 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:58.883492 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:58.907588 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:58.963107 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:58.965889 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:58.998252 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:30:59.018680 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:30:59.019688 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:59.067016 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:59.118394 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:30:59.147909 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:59.162148 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:59.163084 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:59.168856 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:59.172082 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:30:59.205185 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:59.212668 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:30:59.213878 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:30:59.231663 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:30:59.325939 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:30:59.409525 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:30:59.415621 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:30:59.476735 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:30:59.583793 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:59.609443 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:59.616716 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:59.618211 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:59.652086 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:59.701133 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:30:59.751525 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:30:59.874831 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:30:59.884142 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:30:59.891916 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:30:59.908922 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:30:59.927330 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:30:59.937705 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:30:59.961418 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:30:59.990594 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:00.126152 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:00.148363 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:00.149456 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:00.154266 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:00.155684 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:31:00.158315 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:00.208240 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:31:00.289534 13947 raft_consensus.cc:482] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95f32202b9db457fb09872f6f51ea8ba)
I20240502 08:31:00.289785 13947 raft_consensus.cc:504] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:00.290673 13947 leader_election.cc:288] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:31:00.291688 23406 leader_election.cc:341] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Not found: Tablet not found: 6f04a3972e384fa3b0ffe0cb8a4f35aa
W20240502 08:31:00.292228 23404 leader_election.cc:334] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:00.292361 23404 leader_election.cc:302] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b637fbec5444dd91b3af9cdda482f5; no voters: 13e764b27ed449c09aee719cdb2597f8, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:00.292642 13947 raft_consensus.cc:2726] T 6f04a3972e384fa3b0ffe0cb8a4f35aa P 20b637fbec5444dd91b3af9cdda482f5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:00.338675 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:31:00.374464 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:00.383080 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:00.398816 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:00.435842 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:00.440577 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:00.456915 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:00.628887 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:00.810215 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:31:00.839442 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:00.846853 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:00.863694 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:00.866783 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:00.868633 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:00.887326 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:00.908373 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:31:01.007719 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:31:01.118860 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:01.127945 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:01.134173 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:01.149694 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:01.156070 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:01.161162 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:31:01.166271 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:31:01.318001 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:01.358665 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:01.359867 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:01.372783 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:01.390475 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:01.397691 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:01.451400 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:01.482509 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:01.566435 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:01.578681 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:01.620869 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:01.630692 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:01.643151 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:01.644954 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:01.661803 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:01.721946 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:01.747020 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:01.774412 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:01.836402 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:01.847906 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:01.929785 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:02.079030 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.091461 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:02.097390 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:02.105397 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:02.105564 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:02.170353 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:31:02.180006 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:31:02.214071 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:02.306216 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:02.342239 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.349498 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:02.355052 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:02.365837 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:31:02.380975 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:02.395202 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.395620 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:02.542294 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:02.580885 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.590610 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:02.605289 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:02.618981 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.619313 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:02.698395 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:02.859057 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:02.865574 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:02.871171 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:02.886503 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:02.894109 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:02.895738 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:02.928203 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:02.975915 23404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 72056.4us
W20240502 08:31:03.053885 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:03.116006 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:03.276372 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:03.321345 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:31:03.383464 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:03.389489 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:03.392396 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:03.420305 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:03.421181 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:03.620087 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:03.638525 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:03.647035 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:03.647107 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:03.663004 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:31:03.740648 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:03.856134 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:03.885728 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:03.919543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:03.944566 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:03.946971 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:03.950063 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:03.978264 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:03.983299 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:04.001739 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:04.032470 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:04.041368 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:04.160828 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:04.176842 21616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:31:04.207669 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:04.209650 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:04.212445 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:04.215510 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:04.229228 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:04.256080 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 08:31:04.257592 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:04.284276 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:04.337873 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:04.424727 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:04.453271 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:31:04.503301 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:31:04.544831 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:31:04.645777 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:04.671733 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:04.683293 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:04.686352 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:31:04.694798 21617 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:04.697216 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:04.838665 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:04.854374 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:04.931959 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:04.933830 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:04.936414 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:04.944133 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:04.978487 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:05.157495 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:05.159876 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:05.166591 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:05.167507 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:05.167830 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:05.169497 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:05.174890 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:05.327629 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:05.348439 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:05.430526 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:05.436066 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:05.445362 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:05.454277 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:05.460275 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:05.481249 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:05.565883 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:05.591187 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:05.930398 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:05.995765 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:05.998769 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:06.012984 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
************************ BEGIN STACKS **************************
W20240502 08:31:06.060323 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:31:06.093619 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:06.110839 21747 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:06.116415 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:06.121938 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:06.319388 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:06.336119 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:06.346487 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:06.353682 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:06.364624 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:31:06.366175 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:06.378665 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:06.445528 23404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:31:06.463232 23404 consensus_peers.cc:485] T 203e7914da3f4494b239c6603513caee P 20b637fbec5444dd91b3af9cdda482f5 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:31:06.486330 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:06.532598 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:06.584160 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:06.606082 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:06.632637 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:06.646256 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:06.660722 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:06.668619 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:06.686818 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:06.707697 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:06.714093 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:06.754123 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 08:31:06.787501 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:07.026598 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:07.070775 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:07.085736 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:07.093652 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:07.107470 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:31:07.109011 17153 raft_consensus.cc:998] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.585s)
I20240502 08:31:07.109755 17153 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:31:07.121498 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:07.127566 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:31:07.157047 21552 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:31:07.157879 17438 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:31:07.161523 17438 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:07.166775 21552 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:07.173697 21406 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 7 to 8, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:07.200448 17456 raft_consensus.cc:998] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 4b9bfd2541144dad86507451343640d3 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4b9bfd2541144dad86507451343640d3 for more than 300 seconds (300.292s)
I20240502 08:31:07.201217 17456 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:31:07.202298 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:07.208480 23470 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:31:07.209255 17392 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:31:07.224249 17392 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:07.232053 23469 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:07.247953 21407 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 9 to 10, VOTER 4b9bfd2541144dad86507451343640d3 (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:07.272727 21682 consensus_queue.cc:237] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
W20240502 08:31:07.276768 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851): Couldn't send request to peer 13e764b27ed449c09aee719cdb2597f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:07.276748 23469 raft_consensus.cc:1261] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:31:07.277715 17529 consensus_queue.cc:1035] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
[New LWP 8470]
[New LWP 8471]
[New LWP 21373]
[New LWP 21374]
[New LWP 21375]
[New LWP 21376]
[New LWP 21377]
[New LWP 22512]
[New LWP 22513]
[New LWP 22514]
[New LWP 22515]
[New LWP 23920]
I20240502 08:31:07.282207 21813 consensus_queue.cc:237] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:31:07.282310 17529 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:31:07.287928 21552 raft_consensus.cc:1261] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:31:07.288538 21748 consensus_peers.cc:485] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bfc3eaed583458b8381d4e7b4d36aef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:07.288677 17438 consensus_queue.cc:1035] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:31:07.289378 23470 raft_consensus.cc:2932] T d4d933eacd944d84b39f0cb9c45641f8 P 20b637fbec5444dd91b3af9cdda482f5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
W20240502 08:31:07.289953 21396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 08:31:07.290645 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d4d933eacd944d84b39f0cb9c45641f8 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 08:31:07.293288 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2bfc3eaed583458b8381d4e7b4d36aef: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:31:07.293218 21407 catalog_manager.cc:5484] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 10 to 11, VOTER 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
W20240502 08:31:07.293467 21746 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:31:07.298260 17438 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:07.299343 21552 raft_consensus.cc:2932] T 2bfc3eaed583458b8381d4e7b4d36aef P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:07.305426 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2bfc3eaed583458b8381d4e7b4d36aef with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:07.308648 21407 catalog_manager.cc:5484] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:31:07.427795 21616 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:07.451889 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:07.579119 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:07.695767 17566 ts_tablet_manager.cc:902] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:31:07.764011 17566 tablet_copy_client.cc:308] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:31:07.764995 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 13e764b27ed449c09aee719cdb2597f8 ({username='slave'} at 127.8.68.195:60043)
I20240502 08:31:07.765192 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet d4d933eacd944d84b39f0cb9c45641f8 from peer 13e764b27ed449c09aee719cdb2597f8 at {username='slave'} at 127.8.68.195:60043: session id = 13e764b27ed449c09aee719cdb2597f8-d4d933eacd944d84b39f0cb9c45641f8
I20240502 08:31:07.768618 21703 tablet_copy_source_session.cc:205] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:07.776286 17566 ts_tablet_manager.cc:1875] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:31:07.794257 17566 ts_tablet_manager.cc:1888] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:31:07.795302 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d933eacd944d84b39f0cb9c45641f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:07.795797 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:07.796090 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:07.804250 17566 tablet_copy_client.cc:783] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:07.817989 17566 tablet_copy_client.cc:647] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:07.823701 17566 tablet_copy_client.cc:523] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:07.828510 17581 ts_tablet_manager.cc:902] T 2bfc3eaed583458b8381d4e7b4d36aef P 20b637fbec5444dd91b3af9cdda482f5: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:31:07.828933 17566 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap starting.
F20240502 08:31:07.828994 17581 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.queue_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.queue_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_cpu_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_cpu_time_us) (tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_wall_time_us, tablet-download-pool-07f5c2dfcd974ee2b8720e9eb052c3f7.run_wall_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.queue_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.queue_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_cpu_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_cpu_time_us) (tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_wall_time_us, tablet-download-pool-12c17796e5d04014aafdac46984f4ba9.run_wall_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.queue_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.run_cpu_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.run_cpu_time_us) (tablet-download-pool-203e7914da3f4494b239c6603513caee.run_wall_time_us, tablet-download-pool-203e7914da3f4494b239c6603513caee.run_wall_time_us) (tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us, tablet-download-pool-2bfc3eaed583458b8381d4e7b4d36aef.queue_time_us) (tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.queue_time_us, tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.queue_time_us) (tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.run_cpu_time_us, tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.run_cpu_time_us) (tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.run_wall_time_us, tablet-download-pool-3898bfdb9c3d4a05af78f79fd2de61e2.run_wall_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.queue_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.queue_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_cpu_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_cpu_time_us) (tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_wall_time_us, tablet-download-pool-599d5781f1e24ae9ad425a34d88c6b68.run_wall_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.queue_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_cpu_time_us) (tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us, tablet-download-pool-59f5d97e6e0a4755be41699246907e55.run_wall_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.queue_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_cpu_time_us) (tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us, tablet-download-pool-6f04a3972e384fa3b0ffe0cb8a4f35aa.run_wall_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.queue_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.queue_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_cpu_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_cpu_time_us) (tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_wall_time_us, tablet-download-pool-76bb1e3e2b9d4e31bf93ac1ce08a7833.run_wall_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.queue_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.queue_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_cpu_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_cpu_time_us) (tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_wall_time_us, tablet-download-pool-80b3ba1d834a4314b63f336675845441.run_wall_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.queue_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_cpu_time_us) (tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us, tablet-download-pool-d4d933eacd944d84b39f0cb9c45641f8.run_wall_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.queue_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.queue_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_cpu_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_cpu_time_us) (tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_wall_time_us, tablet-download-pool-d8042f6ab6404c5a9f8cce0b8f971542.run_wall_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.queue_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.queue_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_cpu_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_cpu_time_us) (tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_wall_time_us, tablet-download-pool-d96a7174eb5442f2b0cc0cafc3cc4c80.run_wall_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.queue_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.queue_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_cpu_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_cpu_time_us) (tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_wall_time_us, tablet-download-pool-ebf79c69fd614284ba45fd052cac7ebf.run_wall_time_us) (tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.queue_time_us, tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.queue_time_us) (tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.run_cpu_time_us, tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.run_cpu_time_us) (tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.run_wall_time_us, tablet-download-pool-efec2cc693bd4dd992d0bec22fca5419.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7ffa4fb9edcd google::LogMessage::Fail() at ??:0
@ 0x7ffa4fba2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffa4fb9e7cc google::LogMessage::Flush() at ??:0
I20240502 08:31:07.862962 23451 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7ffa4fb9ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffa528df742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:31:07.890367 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7ffa528abf8b kudu::ThreadPool::ThreadPool() at ??:0
W20240502 08:31:07.907192 21616 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
@ 0x7ffa528a5cd6 kudu::ThreadPoolBuilder::Build() at ??:0
W20240502 08:31:07.923790 21877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:40881, user_credentials={real_user=slave}} blocked reactor thread for 43554us
@ 0x7ffa61cf88de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:31:07.932626 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:07.935940 21877 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:07.936853 21749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:46395, user_credentials={real_user=slave}} blocked reactor thread for 62062us
@ 0x7ffa61d10daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ffa61e3b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240502 08:31:07.966866 21746 consensus_peers.cc:462] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
@ 0x7ffa61e73a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240502 08:31:07.986727 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
@ 0x7ffa61e56981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ffa61e5670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa528afb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffa528b319c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ffa528b3019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa5288e01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa55c136db start_thread at ??:0
@ 0x7ffa4ea5171f clone at ??:0
I20240502 08:31:08.039305 17566 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:08.039784 17566 tablet_bootstrap.cc:492] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Bootstrap complete.
I20240502 08:31:08.040094 17566 ts_tablet_manager.cc:1367] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Time spent bootstrapping tablet: real 0.211s user 0.025s sys 0.007s
I20240502 08:31:08.040935 17566 raft_consensus.cc:348] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:08.041149 17566 raft_consensus.cc:729] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e764b27ed449c09aee719cdb2597f8, State: Initialized, Role: FOLLOWER
I20240502 08:31:08.050359 17566 consensus_queue.cc:260] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:08.052327 17566 ts_tablet_manager.cc:1398] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
W20240502 08:31:08.053922 21485 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:08.054628 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session 13e764b27ed449c09aee719cdb2597f8-d4d933eacd944d84b39f0cb9c45641f8 received from {username='slave'} at 127.8.68.195:60043
I20240502 08:31:08.054759 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session 13e764b27ed449c09aee719cdb2597f8-d4d933eacd944d84b39f0cb9c45641f8 on tablet d4d933eacd944d84b39f0cb9c45641f8 with peer 13e764b27ed449c09aee719cdb2597f8
I20240502 08:31:08.144208 21813 raft_consensus.cc:1203] T d4d933eacd944d84b39f0cb9c45641f8 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 08:31:08.239553 21859 maintenance_manager.cc:419] P 13e764b27ed449c09aee719cdb2597f8: Scheduling UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8): 4925 bytes on disk
I20240502 08:31:08.240301 21750 maintenance_manager.cc:643] P 13e764b27ed449c09aee719cdb2597f8: UndoDeltaBlockGCOp(d4d933eacd944d84b39f0cb9c45641f8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 08:31:08.327157 21748 connection.cc:535] client connection to 127.8.68.197:45373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:31:08.327373 21487 connection.cc:535] client connection to 127.8.68.197:45373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:31:08.328253 21879 connection.cc:535] client connection to 127.8.68.197:45373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:31:08.329682 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.329793 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.329880 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.330193 21618 connection.cc:535] client connection to 127.8.68.197:45373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:31:08.330760 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.332473 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:08.332574 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.333016 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.333202 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.333340 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:08.350517 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:08.426997 17588 raft_consensus.cc:998] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.224s)
I20240502 08:31:08.427774 17588 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:31:08.430680 21944 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:31:08.431622 17593 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:31:08.434688 17588 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:08.436142 21944 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:08.441592 21407 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:08.463624 21813 consensus_queue.cc:237] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:31:08.464049 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 852613f14b324b38b5eecc29d4c602cd: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:08.466996 21944 raft_consensus.cc:1261] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:31:08.467727 17588 consensus_queue.cc:1035] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 08:31:08.469691 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:08.470501 17438 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:08.471747 21944 raft_consensus.cc:2932] T 852613f14b324b38b5eecc29d4c602cd P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:08.475317 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 852613f14b324b38b5eecc29d4c602cd with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 08:31:08.476282 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:08.477591 21407 catalog_manager.cc:5484] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 10 to 11, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:31:08.523005 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:08.547631 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:08.565070 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:08.579725 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:08.601778 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:08.623934 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:08.827447 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:08.865228 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:08.878208 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:08.898443 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:08.908649 21746 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:08.911494 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:08.923472 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:08.937085 21485 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:08.955477 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:08.976347 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:31:09.068907 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:09.138195 21616 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:09.172566 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:09.174484 17726 raft_consensus.cc:998] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.336s)
I20240502 08:31:09.175395 17726 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:31:09.203579 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:09.208289 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240502 08:31:09.218284 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:09.225831 21944 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:31:09.226526 17727 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:31:09.234257 21530 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:09.243005 17727 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.246627 21943 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.252893 17438 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:09.253110 17438 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:09.253835 17438 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cfff665685a14d8796aece26addab59c (127.8.68.193:46845), 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373)
I20240502 08:31:09.254384 21407 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 11 to 12, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.256173 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "13e764b27ed449c09aee719cdb2597f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cfff665685a14d8796aece26addab59c" is_pre_election: true
W20240502 08:31:09.265980 21748 leader_election.cc:334] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:09.266145 21748 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8; no voters: 20b637fbec5444dd91b3af9cdda482f5, cfff665685a14d8796aece26addab59c
W20240502 08:31:09.270469 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:31:09.276576 17737 raft_consensus.cc:2726] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:31:09.277268 17737 raft_consensus.cc:998] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.453s)
I20240502 08:31:09.278337 17737 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:31:09.288463 21944 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:31:09.289207 17735 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:31:09.292130 17726 raft_consensus.cc:998] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.169s)
I20240502 08:31:09.292809 17726 consensus_queue.cc:237] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
W20240502 08:31:09.303431 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:09.308634 21944 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:31:09.309978 17190 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:31:09.322219 17610 raft_consensus.cc:998] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.322s)
I20240502 08:31:09.323174 17610 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } }
I20240502 08:31:09.327555 17735 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.335201 21943 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.346586 21410 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.346760 17190 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.348834 17727 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:09.348984 17727 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:09.349773 17727 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:31:09.351992 21943 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
W20240502 08:31:09.355219 21487 leader_election.cc:334] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:09.360178 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8" is_pre_election: true
I20240502 08:31:09.360495 21813 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 2.
I20240502 08:31:09.361148 21487 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:31:09.361466 17726 raft_consensus.cc:2781] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:31:09.361598 17726 raft_consensus.cc:482] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:09.361685 17726 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:31:09.377820 17738 raft_consensus.cc:998] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.165s)
I20240502 08:31:09.378619 17738 consensus_queue.cc:237] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:31:09.380637 21813 consensus_queue.cc:237] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
W20240502 08:31:09.384037 21749 consensus_peers.cc:485] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 -> Peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Couldn't send request to peer 60677121510e47a5b51d1eb22f9aa238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de41d6c6de0400397645c05a0fa17de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.384724 21550 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:31:09.385571 17735 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:31:09.405167 21943 raft_consensus.cc:1261] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:31:09.406178 17737 consensus_queue.cc:1035] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:31:09.413338 17737 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.419682 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2de41d6c6de0400397645c05a0fa17de with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.420471 21404 catalog_manager.cc:5484] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.422627 21410 catalog_manager.cc:5484] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 8 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.435287 21943 raft_consensus.cc:2932] T 2de41d6c6de0400397645c05a0fa17de P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.438681 21552 consensus_queue.cc:237] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:31:09.442462 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:31:09.442569 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:31:09.443595 21944 raft_consensus.cc:1261] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:31:09.444298 17190 consensus_queue.cc:1035] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:31:09.446822 17190 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.449867 21552 consensus_queue.cc:237] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:09.456983 21942 raft_consensus.cc:1261] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:31:09.457587 21944 raft_consensus.cc:2932] T d62f6eec9ee04f93a3738a83f790292c P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.464201 17727 consensus_queue.cc:1035] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:31:09.467576 17190 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.468115 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d62f6eec9ee04f93a3738a83f790292c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.471686 21942 raft_consensus.cc:2932] T 9c0a886b3ba04737a6988c3b58c9698e P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.472620 21407 catalog_manager.cc:5484] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:31:09.481921 21877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 08:31:09.483907 17735 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
W20240502 08:31:09.490057 21488 consensus_peers.cc:485] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c -> Peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Couldn't send request to peer 60677121510e47a5b51d1eb22f9aa238. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.490450 17758 raft_consensus.cc:998] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.400s)
I20240502 08:31:09.491159 17758 consensus_queue.cc:237] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:09.491590 17759 raft_consensus.cc:998] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.298s)
I20240502 08:31:09.491829 21552 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.492301 17759 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:31:09.501322 21944 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:31:09.501797 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9c0a886b3ba04737a6988c3b58c9698e with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.502668 21944 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:31:09.505079 17726 raft_consensus.cc:504] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:09.504984 17744 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:31:09.505878 21407 catalog_manager.cc:5484] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.505870 17726 leader_election.cc:288] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 election: Requested vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:31:09.504984 17758 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 08:31:09.508162 21407 catalog_manager.cc:5484] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 12 to 13, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New cstate: current_term: 1 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.508332 21813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c17796e5d04014aafdac46984f4ba9" candidate_uuid: "cfff665685a14d8796aece26addab59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "13e764b27ed449c09aee719cdb2597f8"
I20240502 08:31:09.508553 21813 raft_consensus.cc:3037] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:31:09.512800 21487 leader_election.cc:334] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:09.513451 21552 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:31:09.514053 17757 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:31:09.514967 21813 raft_consensus.cc:2454] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfff665685a14d8796aece26addab59c in term 3.
I20240502 08:31:09.515467 21487 leader_election.cc:302] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13e764b27ed449c09aee719cdb2597f8, cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5
I20240502 08:31:09.517153 17190 raft_consensus.cc:2781] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:31:09.517385 17727 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:09.517534 17727 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:09.518216 17727 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:09.519799 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d62f6eec9ee04f93a3738a83f790292c: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:09.520153 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:09.521731 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:09.521839 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:31:09.521857 21487 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:09.522219 17726 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:31:09.522505 17190 raft_consensus.cc:686] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [term 3 LEADER]: Becoming Leader. State: Replica: cfff665685a14d8796aece26addab59c, State: Running, Role: LEADER
I20240502 08:31:09.522925 17190 consensus_queue.cc:237] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:09.530794 17758 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.533444 21813 consensus_queue.cc:237] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
W20240502 08:31:09.536623 21749 consensus_peers.cc:485] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Couldn't send request to peer 60677121510e47a5b51d1eb22f9aa238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4862ea2f67c43a28689b464d8ec96e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.536233 21944 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.537179 21552 raft_consensus.cc:1261] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 13e764b27ed449c09aee719cdb2597f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:31:09.537804 17738 consensus_queue.cc:1035] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:31:09.540670 17738 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.544298 21551 raft_consensus.cc:2932] T c4862ea2f67c43a28689b464d8ec96e6 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.546564 21397 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c4862ea2f67c43a28689b464d8ec96e6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.546922 21404 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 7 to 8, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } }
I20240502 08:31:09.550308 21410 catalog_manager.cc:5484] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 6 to 7, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New cstate: current_term: 1 leader_uuid: "13e764b27ed449c09aee719cdb2597f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.553880 21682 consensus_queue.cc:237] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
W20240502 08:31:09.557526 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.560562 21944 raft_consensus.cc:1261] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:31:09.562680 17758 consensus_queue.cc:1035] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:31:09.566056 17745 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.566673 21942 raft_consensus.cc:2932] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.566655 21407 catalog_manager.cc:5484] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c reported cstate change: term changed from 2 to 3, leader changed from 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) to cfff665685a14d8796aece26addab59c (127.8.68.193). New cstate: current_term: 3 leader_uuid: "cfff665685a14d8796aece26addab59c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:31:09.570792 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8a45b1da348e4bb89f021df103a0603f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.573907 17759 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.573889 21410 catalog_manager.cc:5484] T 8a45b1da348e4bb89f021df103a0603f P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 8 to 9, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.577414 17745 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.578467 21943 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.582181 21550 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.584072 21410 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b reported cstate change: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:09.594123 21407 catalog_manager.cc:5484] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 5 to 6, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.599015 21943 consensus_queue.cc:237] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:09.603962 21682 consensus_queue.cc:237] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } }
I20240502 08:31:09.608287 21944 raft_consensus.cc:1261] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:31:09.610529 17744 consensus_queue.cc:1035] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:31:09.611661 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.614107 17745 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
W20240502 08:31:09.615490 21880 consensus_peers.cc:485] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745): Couldn't send request to peer 60677121510e47a5b51d1eb22f9aa238. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.616644 21550 raft_consensus.cc:1261] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Refusing update from remote peer 0f655ab8f20d4ba482c3173f9594b98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:31:09.617305 17757 consensus_queue.cc:1035] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:31:09.619000 21944 raft_consensus.cc:2932] T 1648586b889b4a1e9ac34481cec6257a P 0f655ab8f20d4ba482c3173f9594b98b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.619352 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1648586b889b4a1e9ac34481cec6257a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.620628 17757 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.623876 21551 raft_consensus.cc:2932] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
I20240502 08:31:09.625047 21407 catalog_manager.cc:5484] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 reported cstate change: config changed from index 6 to 7, VOTER 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } }
I20240502 08:31:09.632954 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cec56facd7f740aeb42bc7c32495dae3 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:09.638917 21407 catalog_manager.cc:5484] T cec56facd7f740aeb42bc7c32495dae3 P cfff665685a14d8796aece26addab59c reported cstate change: config changed from index 10 to 11, VOTER 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194) added. New cstate: current_term: 1 leader_uuid: "0f655ab8f20d4ba482c3173f9594b98b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } }
W20240502 08:31:09.676018 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:31:09.786789 17797 ts_tablet_manager.cc:902] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
W20240502 08:31:09.794888 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:09.835216 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:09.865162 17797 tablet_copy_client.cc:308] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:31:09.866397 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet 2de41d6c6de0400397645c05a0fa17de from peer 60677121510e47a5b51d1eb22f9aa238 ({username='slave'} at 127.8.68.194:40215)
I20240502 08:31:09.866597 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet 2de41d6c6de0400397645c05a0fa17de from peer 60677121510e47a5b51d1eb22f9aa238 at {username='slave'} at 127.8.68.194:40215: session id = 60677121510e47a5b51d1eb22f9aa238-2de41d6c6de0400397645c05a0fa17de
I20240502 08:31:09.870343 21833 tablet_copy_source_session.cc:205] T 2de41d6c6de0400397645c05a0fa17de P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:09.874600 17797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de41d6c6de0400397645c05a0fa17de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:09.901374 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:09.925310 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:09.929755 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:09.961637 17797 tablet_copy_client.cc:783] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:09.964558 17808 ts_tablet_manager.cc:902] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Initiating tablet copy from peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845)
W20240502 08:31:09.975945 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:09.978155 17797 tablet_copy_client.cc:647] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:09.987874 17797 tablet_copy_client.cc:523] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:10.005379 21813 raft_consensus.cc:1261] T 12c17796e5d04014aafdac46984f4ba9 P 13e764b27ed449c09aee719cdb2597f8 [term 3 FOLLOWER]: Refusing update from remote peer cfff665685a14d8796aece26addab59c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:31:10.006356 17190 consensus_queue.cc:1035] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:31:10.020145 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:10.023509 17811 ts_tablet_manager.cc:902] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Initiating tablet copy from peer 13e764b27ed449c09aee719cdb2597f8 (127.8.68.195:33851)
I20240502 08:31:10.050966 17797 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:31:10.068325 17808 tablet_copy_client.cc:308] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:46845
I20240502 08:31:10.069200 21572 tablet_copy_service.cc:140] P cfff665685a14d8796aece26addab59c: Received BeginTabletCopySession request for tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 60677121510e47a5b51d1eb22f9aa238 ({username='slave'} at 127.8.68.194:51169)
I20240502 08:31:10.069384 21572 tablet_copy_service.cc:161] P cfff665685a14d8796aece26addab59c: Beginning new tablet copy session on tablet 9c0a886b3ba04737a6988c3b58c9698e from peer 60677121510e47a5b51d1eb22f9aa238 at {username='slave'} at 127.8.68.194:51169: session id = 60677121510e47a5b51d1eb22f9aa238-9c0a886b3ba04737a6988c3b58c9698e
I20240502 08:31:10.073695 21572 tablet_copy_source_session.cc:205] T 9c0a886b3ba04737a6988c3b58c9698e P cfff665685a14d8796aece26addab59c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:10.074995 17808 ts_tablet_manager.cc:1875] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:31:10.083756 17808 ts_tablet_manager.cc:1888] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:31:10.084050 17811 tablet_copy_client.cc:308] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:33851
I20240502 08:31:10.084388 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0a886b3ba04737a6988c3b58c9698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:10.085234 21833 tablet_copy_service.cc:140] P 13e764b27ed449c09aee719cdb2597f8: Received BeginTabletCopySession request for tablet c4862ea2f67c43a28689b464d8ec96e6 from peer 60677121510e47a5b51d1eb22f9aa238 ({username='slave'} at 127.8.68.194:40215)
I20240502 08:31:10.085408 21833 tablet_copy_service.cc:161] P 13e764b27ed449c09aee719cdb2597f8: Beginning new tablet copy session on tablet c4862ea2f67c43a28689b464d8ec96e6 from peer 60677121510e47a5b51d1eb22f9aa238 at {username='slave'} at 127.8.68.194:40215: session id = 60677121510e47a5b51d1eb22f9aa238-c4862ea2f67c43a28689b464d8ec96e6
I20240502 08:31:10.089249 21833 tablet_copy_source_session.cc:205] T c4862ea2f67c43a28689b464d8ec96e6 P 13e764b27ed449c09aee719cdb2597f8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:10.090783 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4862ea2f67c43a28689b464d8ec96e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:10.093384 17808 tablet_copy_client.cc:783] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:10.100731 17808 tablet_copy_client.cc:647] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:10.102586 17808 tablet_copy_client.cc:523] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:10.102787 17811 tablet_copy_client.cc:783] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:10.108839 17811 tablet_copy_client.cc:647] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:10.108863 17808 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:31:10.112946 17811 tablet_copy_client.cc:523] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:10.119907 17811 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:31:10.143188 17797 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:10.143880 17797 tablet_bootstrap.cc:492] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Bootstrap complete.
I20240502 08:31:10.144590 17797 ts_tablet_manager.cc:1367] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.094s user 0.025s sys 0.001s
I20240502 08:31:10.145560 17797 raft_consensus.cc:348] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.145817 17797 raft_consensus.cc:729] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:31:10.146143 17797 consensus_queue.cc:260] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.150099 17797 ts_tablet_manager.cc:1398] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:31:10.153883 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 60677121510e47a5b51d1eb22f9aa238-2de41d6c6de0400397645c05a0fa17de received from {username='slave'} at 127.8.68.194:40215
I20240502 08:31:10.154008 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 60677121510e47a5b51d1eb22f9aa238-2de41d6c6de0400397645c05a0fa17de on tablet 2de41d6c6de0400397645c05a0fa17de with peer 60677121510e47a5b51d1eb22f9aa238
I20240502 08:31:10.158135 17808 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:10.158735 17808 tablet_bootstrap.cc:492] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Bootstrap complete.
I20240502 08:31:10.159142 17808 ts_tablet_manager.cc:1367] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.011s
I20240502 08:31:10.160241 17808 raft_consensus.cc:348] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.160609 17808 raft_consensus.cc:729] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:31:10.160989 17808 consensus_queue.cc:260] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.161840 17808 ts_tablet_manager.cc:1398] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:31:10.162617 21572 tablet_copy_service.cc:342] P cfff665685a14d8796aece26addab59c: Request end of tablet copy session 60677121510e47a5b51d1eb22f9aa238-9c0a886b3ba04737a6988c3b58c9698e received from {username='slave'} at 127.8.68.194:51169
I20240502 08:31:10.162755 21572 tablet_copy_service.cc:433] P cfff665685a14d8796aece26addab59c: ending tablet copy session 60677121510e47a5b51d1eb22f9aa238-9c0a886b3ba04737a6988c3b58c9698e on tablet 9c0a886b3ba04737a6988c3b58c9698e with peer 60677121510e47a5b51d1eb22f9aa238
I20240502 08:31:10.165040 21682 raft_consensus.cc:1203] T 2de41d6c6de0400397645c05a0fa17de P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:31:10.196426 17811 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:10.196923 17811 tablet_bootstrap.cc:492] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap complete.
I20240502 08:31:10.197010 17808 ts_tablet_manager.cc:902] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Initiating tablet copy from peer 0f655ab8f20d4ba482c3173f9594b98b (127.8.68.196:46395)
I20240502 08:31:10.197276 17811 ts_tablet_manager.cc:1367] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.077s user 0.040s sys 0.008s
I20240502 08:31:10.198309 17811 raft_consensus.cc:348] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.198585 17811 raft_consensus.cc:729] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:31:10.199030 17811 consensus_queue.cc:260] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.199806 17811 ts_tablet_manager.cc:1398] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:31:10.200554 21833 tablet_copy_service.cc:342] P 13e764b27ed449c09aee719cdb2597f8: Request end of tablet copy session 60677121510e47a5b51d1eb22f9aa238-c4862ea2f67c43a28689b464d8ec96e6 received from {username='slave'} at 127.8.68.194:40215
I20240502 08:31:10.200680 21833 tablet_copy_service.cc:433] P 13e764b27ed449c09aee719cdb2597f8: ending tablet copy session 60677121510e47a5b51d1eb22f9aa238-c4862ea2f67c43a28689b464d8ec96e6 on tablet c4862ea2f67c43a28689b464d8ec96e6 with peer 60677121510e47a5b51d1eb22f9aa238
I20240502 08:31:10.204551 17808 tablet_copy_client.cc:308] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:46395
I20240502 08:31:10.205145 21964 tablet_copy_service.cc:140] P 0f655ab8f20d4ba482c3173f9594b98b: Received BeginTabletCopySession request for tablet cec56facd7f740aeb42bc7c32495dae3 from peer 60677121510e47a5b51d1eb22f9aa238 ({username='slave'} at 127.8.68.194:43991)
I20240502 08:31:10.205291 21964 tablet_copy_service.cc:161] P 0f655ab8f20d4ba482c3173f9594b98b: Beginning new tablet copy session on tablet cec56facd7f740aeb42bc7c32495dae3 from peer 60677121510e47a5b51d1eb22f9aa238 at {username='slave'} at 127.8.68.194:43991: session id = 60677121510e47a5b51d1eb22f9aa238-cec56facd7f740aeb42bc7c32495dae3
I20240502 08:31:10.208048 21964 tablet_copy_source_session.cc:205] T cec56facd7f740aeb42bc7c32495dae3 P 0f655ab8f20d4ba482c3173f9594b98b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:10.209161 17808 ts_tablet_manager.cc:1875] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:31:10.216205 17808 ts_tablet_manager.cc:1888] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:31:10.216447 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec56facd7f740aeb42bc7c32495dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:10.220393 17808 tablet_copy_client.cc:783] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:10.227913 17808 tablet_copy_client.cc:647] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:10.229921 17808 tablet_copy_client.cc:523] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:10.234905 17808 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap starting.
I20240502 08:31:10.257177 17808 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:10.257611 17808 tablet_bootstrap.cc:492] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Bootstrap complete.
I20240502 08:31:10.257898 17808 ts_tablet_manager.cc:1367] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.003s
I20240502 08:31:10.258641 17808 raft_consensus.cc:348] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.258886 17808 raft_consensus.cc:729] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60677121510e47a5b51d1eb22f9aa238, State: Initialized, Role: FOLLOWER
I20240502 08:31:10.259172 17808 consensus_queue.cc:260] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0f655ab8f20d4ba482c3173f9594b98b" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 46395 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } }
I20240502 08:31:10.260671 17808 ts_tablet_manager.cc:1398] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:31:10.262670 21964 tablet_copy_service.cc:342] P 0f655ab8f20d4ba482c3173f9594b98b: Request end of tablet copy session 60677121510e47a5b51d1eb22f9aa238-cec56facd7f740aeb42bc7c32495dae3 received from {username='slave'} at 127.8.68.194:43991
I20240502 08:31:10.262778 21964 tablet_copy_service.cc:433] P 0f655ab8f20d4ba482c3173f9594b98b: ending tablet copy session 60677121510e47a5b51d1eb22f9aa238-cec56facd7f740aeb42bc7c32495dae3 on tablet cec56facd7f740aeb42bc7c32495dae3 with peer 60677121510e47a5b51d1eb22f9aa238
W20240502 08:31:10.292906 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:10.318344 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de): 4925 bytes on disk
I20240502 08:31:10.319080 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(2de41d6c6de0400397645c05a0fa17de) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:31:10.321429 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e): 4840 bytes on disk
I20240502 08:31:10.324527 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(9c0a886b3ba04737a6988c3b58c9698e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 08:31:10.325914 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3): 4840 bytes on disk
I20240502 08:31:10.327126 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(cec56facd7f740aeb42bc7c32495dae3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 08:31:10.328191 21729 maintenance_manager.cc:419] P 60677121510e47a5b51d1eb22f9aa238: Scheduling UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6): 4586 bytes on disk
I20240502 08:31:10.328631 21620 maintenance_manager.cc:643] P 60677121510e47a5b51d1eb22f9aa238: UndoDeltaBlockGCOp(c4862ea2f67c43a28689b464d8ec96e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 08:31:10.391322 21682 raft_consensus.cc:1203] T 9c0a886b3ba04737a6988c3b58c9698e P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240502 08:31:10.398979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:10.399974 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:10.422288 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:10.444886 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:10.450892 21682 raft_consensus.cc:1203] T c4862ea2f67c43a28689b464d8ec96e6 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 08:31:10.568576 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2de41d6c6de0400397645c05a0fa17de: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:10.610874 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:10.658975 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:10.685847 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:10.704162 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:10.712291 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:10.718940 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:10.733121 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 08:31:10.734133 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:10.746640 21682 raft_consensus.cc:1203] T cec56facd7f740aeb42bc7c32495dae3 P 60677121510e47a5b51d1eb22f9aa238 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240502 08:31:10.767928 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:10.768085 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:10.783067 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:10.790946 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:10.849290 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:10.864758 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:10.893409 17745 raft_consensus.cc:998] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Attempting to remove follower 95f32202b9db457fb09872f6f51ea8ba from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95f32202b9db457fb09872f6f51ea8ba for more than 300 seconds (300.107s)
I20240502 08:31:10.894166 17745 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } }
I20240502 08:31:10.897013 21813 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:31:10.897727 17745 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:31:10.911993 17914 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:31:10.912720 21812 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
I20240502 08:31:10.920048 21411 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 9 to 10, VOTER 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } }
W20240502 08:31:10.941612 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:10.954813 21682 consensus_queue.cc:237] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
W20240502 08:31:10.962870 21619 consensus_peers.cc:485] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 -> Peer cfff665685a14d8796aece26addab59c (127.8.68.193:46845): Couldn't send request to peer cfff665685a14d8796aece26addab59c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b88c4a1a36424086aad89b7585c17960. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:31:10.962898 21813 raft_consensus.cc:1261] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Refusing update from remote peer 60677121510e47a5b51d1eb22f9aa238: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:31:10.963752 17912 consensus_queue.cc:1035] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:31:10.966717 17914 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:10.967378 21813 raft_consensus.cc:2932] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
I20240502 08:31:10.973218 21398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b88c4a1a36424086aad89b7585c17960 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:31:10.973920 21407 catalog_manager.cc:5484] T b88c4a1a36424086aad89b7585c17960 P 13e764b27ed449c09aee719cdb2597f8 reported cstate change: config changed from index 10 to 11, VOTER cfff665685a14d8796aece26addab59c (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "60677121510e47a5b51d1eb22f9aa238" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } }
W20240502 08:31:11.012454 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:11.024549 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:11.047575 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:11.067011 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:11.099704 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:11.102102 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:11.124451 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:11.125258 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:11.267535 21747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 08:31:11.315277 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:31:11.385499 17954 ts_tablet_manager.cc:902] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Initiating tablet copy from peer 60677121510e47a5b51d1eb22f9aa238 (127.8.68.194:36745)
I20240502 08:31:11.388749 17954 tablet_copy_client.cc:308] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:36745
I20240502 08:31:11.389588 21703 tablet_copy_service.cc:140] P 60677121510e47a5b51d1eb22f9aa238: Received BeginTabletCopySession request for tablet b88c4a1a36424086aad89b7585c17960 from peer cfff665685a14d8796aece26addab59c ({username='slave'} at 127.8.68.193:47687)
I20240502 08:31:11.389767 21703 tablet_copy_service.cc:161] P 60677121510e47a5b51d1eb22f9aa238: Beginning new tablet copy session on tablet b88c4a1a36424086aad89b7585c17960 from peer cfff665685a14d8796aece26addab59c at {username='slave'} at 127.8.68.193:47687: session id = cfff665685a14d8796aece26addab59c-b88c4a1a36424086aad89b7585c17960
I20240502 08:31:11.393435 21703 tablet_copy_source_session.cc:205] T b88c4a1a36424086aad89b7585c17960 P 60677121510e47a5b51d1eb22f9aa238: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:31:11.394711 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88c4a1a36424086aad89b7585c17960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:31:11.400368 17954 tablet_copy_client.cc:783] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 5 data blocks...
I20240502 08:31:11.406646 17954 tablet_copy_client.cc:647] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:31:11.408526 17954 tablet_copy_client.cc:523] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:31:11.413647 17954 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Bootstrap starting.
W20240502 08:31:11.421943 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:11.427505 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 08:31:11.433722 17954 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:31:11.434201 17954 tablet_bootstrap.cc:492] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Bootstrap complete.
I20240502 08:31:11.434535 17954 ts_tablet_manager.cc:1367] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20240502 08:31:11.435199 17954 raft_consensus.cc:348] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:31:11.435480 17954 raft_consensus.cc:729] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff665685a14d8796aece26addab59c, State: Initialized, Role: FOLLOWER
I20240502 08:31:11.435827 17954 consensus_queue.cc:260] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "60677121510e47a5b51d1eb22f9aa238" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 36745 } } peers { permanent_uuid: "13e764b27ed449c09aee719cdb2597f8" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 33851 } } peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } }
I20240502 08:31:11.437170 17954 ts_tablet_manager.cc:1398] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:31:11.438812 21703 tablet_copy_service.cc:342] P 60677121510e47a5b51d1eb22f9aa238: Request end of tablet copy session cfff665685a14d8796aece26addab59c-b88c4a1a36424086aad89b7585c17960 received from {username='slave'} at 127.8.68.193:47687
I20240502 08:31:11.438957 21703 tablet_copy_service.cc:433] P 60677121510e47a5b51d1eb22f9aa238: ending tablet copy session cfff665685a14d8796aece26addab59c-b88c4a1a36424086aad89b7585c17960 on tablet b88c4a1a36424086aad89b7585c17960 with peer cfff665685a14d8796aece26addab59c
I20240502 08:31:11.439668 17726 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:11.439838 17726 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:11.440510 17726 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:11.443054 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:11.443500 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:11.443640 21487 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:11.443929 17726 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:11.549027 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:11.569674 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:11.583710 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:11.595045 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:11.595942 21598 maintenance_manager.cc:419] P cfff665685a14d8796aece26addab59c: Scheduling UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960): 4586 bytes on disk
I20240502 08:31:11.596747 21489 maintenance_manager.cc:643] P cfff665685a14d8796aece26addab59c: UndoDeltaBlockGCOp(b88c4a1a36424086aad89b7585c17960) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240502 08:31:11.638654 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:11.683107 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:11.710180 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:11.806301 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:31:11.806504 21551 raft_consensus.cc:1203] T b88c4a1a36424086aad89b7585c17960 P cfff665685a14d8796aece26addab59c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240502 08:31:11.846524 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:31:11.863497 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:11.994326 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:11.999089 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:12.014519 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:12.015545 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:12.025619 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:12.039172 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:12.069253 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:31:12.096580 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:12.258955 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:12.283334 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:12.283324 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:12.291971 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:12.402643 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:31:12.544202 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:12.550956 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 08:31:12.561630 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:12.568462 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:12.571638 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:12.598853 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:12.741937 21395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 08:31:12.743152 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 852613f14b324b38b5eecc29d4c602cd: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:13.035629 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:13.043584 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:13.068357 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:13.080051 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:13.092842 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.116647 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.134408 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.159294 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:13.192919 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:13.243292 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.255512 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.277067 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.295323 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:13.334367 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:13.340785 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:13.354811 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:13.362855 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:13.372054 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:13.478454 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.507032 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.516343 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:13.551398 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:31:13.575364 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:13.584635 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:13.604501 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:13.612252 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:13.619153 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:13.773826 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:13.776880 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 08:31:13.791592 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d96a7174eb5442f2b0cc0cafc3cc4c80: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:31:13.915307 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:13.940469 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:13.971206 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:13.973400 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:13.988832 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:14.010449 18147 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:14.010646 18147 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:14.011407 18147 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:14.014377 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:14.014628 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:14.014708 21485 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:14.015151 18147 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:14.072199 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:14.091032 21400 maintenance_manager.cc:419] P 8d832776683f415c8e7f1a1bc071aed9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2460 bytes on disk
I20240502 08:31:14.095674 21399 maintenance_manager.cc:643] P 8d832776683f415c8e7f1a1bc071aed9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.000s Metrics: {"cfile_init":1,"dirs.queue_time_us":363,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":120,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"thread_start_us":502,"threads_started":2,"wal-append.queue_time_us":358}
W20240502 08:31:14.237090 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 08:31:14.239153 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:14.278205 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:31:14.279414 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:14.308657 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:14.337620 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:31:14.400645 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:31:14.447872 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:14.468257 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:14.489354 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:14.499289 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:14.511569 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:14.621233 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:31:14.703449 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:31:14.736696 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:14.745752 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:14.773310 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:14.788841 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:14.821897 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:31:14.844619 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:31:14.861523 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:14.876148 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 08:31:15.030084 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:15.051609 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:15.056823 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:15.060796 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:15.231073 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:15.333824 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:15.376632 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:15.413125 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:15.448957 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:15.462124 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:15.557897 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.597458 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.624308 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.701280 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.763687 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.789005 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:15.827425 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.870175 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.905390 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.935453 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.956323 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:15.976384 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:31:15.978511 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:16.004493 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:16.015442 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:16.020013 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:31:16.041473 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:16.056954 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:16.066169 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:16.264351 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:16.268934 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:31:16.269917 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 08:31:16.270247 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:16.274735 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:16.294488 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:16.483517 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:16.508528 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:16.509687 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 08:31:16.531220 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:16.542505 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:16.547031 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:16.609714 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:16.730789 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 08:31:16.778540 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:16.784013 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:16.805791 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:16.832374 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:31:16.839504 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 08:31:16.873411 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:16.903381 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 852613f14b324b38b5eecc29d4c602cd: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:16.924085 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:16.960639 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:17.081141 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:31:17.134955 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 08:31:17.177618 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:31:17.290016 18567 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:17.290230 18567 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:17.291229 18567 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:17.296761 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:17.298303 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:17.298444 21487 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:17.300560 18567 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:17.323011 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 08:31:17.389321 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:17.389873 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:17.397713 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:17.405563 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:31:17.431329 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:17.498281 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:17.659339 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:17.661197 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:17.702838 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:17.739598 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:17.821321 21395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 08:31:17.953917 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9c0a886b3ba04737a6988c3b58c9698e: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:17.992808 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.000667 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.013768 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:31:18.047981 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:18.063513 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:18.086200 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:18.100940 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.101334 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:18.101502 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:18.121879 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.190647 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.249627 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:31:18.336943 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:18.356206 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:18.362622 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:18.385341 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:18.467005 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.485898 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.518369 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.553002 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:31:18.561940 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.584746 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:18.609795 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:18.635891 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:18.830531 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 08:31:18.836421 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:18.867803 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:18.976965 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:18.982164 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:19.006706 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:19.044814 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:19.205610 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:19.214483 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:31:19.222182 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:19.231176 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:19.233392 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:19.276899 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:31:19.336760 21485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:31:19.337906 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 08:31:19.355005 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:19.359953 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:19.410979 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:19.432166 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:31:19.470681 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:19.481421 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:19.502035 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:19.518620 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:19.522601 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 08:31:19.525542 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:19.575134 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 08:31:19.618518 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:19.672353 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 08:31:19.749650 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:19.764444 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:31:19.770170 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:19.779678 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:19.782855 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:19.844535 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:31:20.107299 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:20.307116 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:20.332108 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:20.349658 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:20.355572 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:20.386584 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.400041 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.506805 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.544030 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.613188 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:20.622056 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:20.629184 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:31:20.636260 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:20.638250 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:20.666774 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:20.732518 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.790653 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.890326 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:20.901929 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:31:20.952368 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:20.981738 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:20.995236 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:21.025895 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:21.026001 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:21.045534 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:21.086527 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:21.120976 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:21.220268 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 08:31:21.248837 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:21.253888 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:21.255000 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:21.257447 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 08:31:21.274542 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:21.297245 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:31:21.566265 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:21.566962 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:21.664966 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:21.737789 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 08:31:21.747892 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:21.748337 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:21.768599 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:21.793684 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:21.809099 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:21.815395 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:21.887342 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:21.892918 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:21.893375 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:22.009313 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:31:22.068894 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:22.141669 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:22.158147 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:31:22.178730 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:22.189795 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:22.205644 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:22.211764 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:22.222967 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 08:31:22.379456 18567 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:22.379666 18567 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:22.380367 18567 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:22.383662 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:22.385823 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:22.385954 21485 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:22.388260 18567 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:31:22.420332 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:22.429487 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:22.459309 21618 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:33851, user_credentials={real_user=slave}} blocked reactor thread for 49670.5us
I20240502 08:31:22.472625 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:22.494194 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:22.671053 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:22.730206 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:22.746892 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:22.759467 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:22.800168 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:22.863737 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.023013 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.049409 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.110354 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.143420 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.156358 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.241875 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:23.278703 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:23.281476 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:23.302217 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:23.319561 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:23.324918 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.466632 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.473811 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.512555 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:31:23.520817 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:23.540462 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:23.553109 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:23.569084 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:23.645284 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:23.671586 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:23.691901 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:23.775549 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:23.793125 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:23.796825 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:31:23.801681 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:23.807502 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:23.820866 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:23.853911 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 08:31:23.878661 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:23.975759 21396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240502 08:31:23.977764 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d4d933eacd944d84b39f0cb9c45641f8: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:31:24.132946 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:24.155110 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:24.168455 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:24.181285 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:24.207329 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:24.241621 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:24.374670 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:31:24.375931 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:24.403271 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 08:31:24.422463 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:24.489426 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:24.516824 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 08:31:24.535589 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:24.578197 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:24.615499 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 08:31:24.644415 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:24.646034 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:24.688818 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:24.689518 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:31:24.694356 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:24.713245 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 08:31:24.719064 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:31:24.759555 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:25.056066 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:25.066839 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:25.091324 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:25.105583 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:25.157816 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 852613f14b324b38b5eecc29d4c602cd: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:25.209980 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:25.308189 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:25.320958 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:25.350553 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:25.351534 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20240502 08:31:25.493482 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.525161 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.537464 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:31:25.630543 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:25.669302 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:25.674835 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:31:25.696823 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:25.700002 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:25.764879 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.796150 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.828682 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:25.851007 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.910575 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.962119 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:25.989601 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:26.081952 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:31:26.147169 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:26.149152 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:26.172266 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:26.189932 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c4862ea2f67c43a28689b464d8ec96e6: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:26.211727 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:26.221661 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:26.362813 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:31:26.364637 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:26.393546 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:26.405488 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 08:31:26.450017 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:26.467391 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 08:31:26.477327 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:26.481598 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:26.482271 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:26.485661 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:26.633889 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:26.646229 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 08:31:26.712209 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:26.726045 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:26.741518 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:26.776229 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:26.913081 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:26.929975 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:26.930496 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 08:31:27.085444 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 08:31:27.094558 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:27.108086 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:31:27.134723 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:31:27.137862 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:27.159199 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:31:27.168865 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:27.171002 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:27.188995 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:27.190017 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:27.291875 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:27.576045 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:27.611056 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:27.618324 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:27.625991 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:27.655090 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:27.667588 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:27.926589 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:27.973445 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.005045 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:28.009142 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:28.028012 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:28.033986 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:28.044755 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:28.055742 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:28.055905 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.276453 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:28.298107 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.307786 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:28.315568 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:28.341512 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.343470 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:28.435314 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:28.439661 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:28.453372 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:28.481413 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:28.519484 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.585407 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:28.627586 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:28.634303 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:28.638466 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:28.668401 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:28.751681 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:28.801106 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:28.838212 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:28.903136 18567 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:28.903448 18567 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:28.904304 18567 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:28.906118 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:28.906520 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:28.906656 21487 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:28.906985 18567 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:28.967675 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:29.038067 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:29.065517 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:29.112928 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:29.142470 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:29.170858 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:29.172538 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:29.255625 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:29.313115 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:29.412921 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 08:31:29.420958 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:29.439181 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 08:31:29.442287 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:29.444109 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:29.452015 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:29.463925 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 08:31:29.465766 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 08:31:29.471691 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:29.584206 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:29.707047 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:29.714570 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 08:31:29.723973 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:29.723979 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:29.745191 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:31:29.752341 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:29.760752 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:29.821540 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:31:29.890678 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:29.916839 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 08:31:30.064318 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:30.075011 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:30.109100 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:30.117697 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:30.121256 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:30.460968 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.532398 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:31:30.566877 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:30.577450 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:30.618713 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:30.629091 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:30.639595 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:30.698711 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.737455 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.867861 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.877691 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.885314 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.906833 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.950011 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:30.958058 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:31:30.964124 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:30.997325 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:31.019553 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:31.024137 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:31.079172 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:31.089259 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:31.197702 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 08:31:31.242061 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:31.256886 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:31:31.257158 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:31.278409 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:31.304507 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:31.346904 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:31.382211 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:31:31.416923 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:31.567445 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:31.582091 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:31.582792 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:31.583698 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:31.602219 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:31.612470 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:31.756068 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:31.819590 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:31.995323 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:32.033355 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:32.074992 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:32.075922 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 08:31:32.123703 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:32.139215 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:32.165390 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:32.184847 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:32.205633 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:32.277846 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:32.377108 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:31:32.491050 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 08:31:32.502936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:32.503964 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:32.507037 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:32.508651 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:32.570209 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:32.705603 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:32.778893 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:32.809943 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:32.817366 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:32.825510 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:32.826628 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:33.096742 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:33.141000 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:33.142097 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:33.145404 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:33.175499 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:33.196475 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:33.202900 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.219923 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.273171 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.289040 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.314456 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.606096 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:33.628175 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:33.661484 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:33.669476 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:33.673671 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:33.698835 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.721560 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:33.723331 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.796913 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:33.813951 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:33.915306 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:33.920655 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:33.960624 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:34.016170 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:34.016436 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:34.034278 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:34.053411 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:34.136211 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 08:31:34.246461 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:34.247568 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:34.261582 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:34.290220 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 08:31:34.306655 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:34.307262 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:34.413997 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:31:34.419843 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:34.438495 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:34.486377 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:31:34.505874 21618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 08:31:34.506914 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:34.569604 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:34.573408 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:34.593933 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:34.624614 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:34.630499 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:34.638657 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:34.692131 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 08:31:34.704237 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:34.924723 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:31:35.059165 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:35.066998 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:35.083755 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:35.089232 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:35.164431 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:35.202237 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 08:31:35.283700 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:35.337663 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:31:35.365864 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:35.404301 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:35.430896 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:35.449450 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:35.540769 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:35.593043 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:31:35.685504 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:35.711494 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:35.750694 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:35.758559 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:31:35.775911 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:35.842505 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:35.891068 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:35.897125 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:35.915992 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:31:36.025665 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:36.050144 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:36.068610 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:36.070315 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:36.086238 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.130537 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.145026 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.208518 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:36.254540 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.360383 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.444351 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 08:31:36.445466 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:36.456326 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:36.484679 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:36.513653 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 08:31:36.552076 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:36.556385 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:36.603453 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:36.604748 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:36.771162 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 08:31:36.836244 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:36.949503 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 08:31:36.995741 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:36.996044 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 08:31:37.010476 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:37.017947 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:37.053606 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:37.068727 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:37.149827 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:37.151001 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 08:31:37.159370 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:31:37.246083 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 08:31:37.314334 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:31:37.466511 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:37.502671 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:37.503160 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:37.508458 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:37.587553 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:37.592393 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:37.781584 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:37.784968 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:37.799999 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:37.813256 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:37.815116 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:38.027828 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.084566 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:38.174742 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.211309 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.225890 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:31:38.370585 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:38.391886 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:38.392414 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:38.406139 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:38.425557 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.505132 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.556730 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.563937 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:38.669742 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:38.671964 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:38.687763 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:31:38.689167 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:38.730427 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:38.731468 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:38.751669 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:38.840034 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:39.026841 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 08:31:39.034533 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:39.048950 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 08:31:39.053498 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:39.065235 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:39.080121 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:39.093421 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 08:31:39.123700 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 08:31:39.154208 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:39.159121 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:39.208070 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:31:39.306041 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:39.331390 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:39.351336 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:39.393082 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:39.495218 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 08:31:39.517081 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 08:31:39.528638 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:39.603219 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:31:39.604341 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:31:39.637050 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:39.638413 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:39.746189 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 08:31:39.848632 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:39.857967 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 08:31:39.891649 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:31:39.892803 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:39.947002 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:39.965699 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:40.013509 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:40.128697 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:40.135782 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:31:40.215484 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:40.232985 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:40.238953 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:40.258318 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:40.365829 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 08:31:40.367491 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:40.408308 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:40.410459 21396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:31:40.411516 21396 catalog_manager.cc:4573] TS 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d4d933eacd944d84b39f0cb9c45641f8: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111)
I20240502 08:31:40.498651 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:40.514370 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:40.535965 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:40.539433 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:40.618621 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:31:40.785109 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:40.824299 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:40.837999 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:40.846459 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:31:40.899920 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:40.907107 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:40.921157 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:40.925128 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:40.937898 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:40.940157 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:40.995414 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:41.337452 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 08:31:41.352635 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:41.360661 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:41.384280 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:41.401710 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:41.402740 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:41.455559 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:31:41.456919 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 08:31:41.482743 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:41.508134 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:41.573958 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:41.589954 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 08:31:41.592415 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 852613f14b324b38b5eecc29d4c602cd: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:41.645637 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 08:31:41.669289 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:41.681998 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:41.693089 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 08:31:41.718475 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:41.733755 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:41.795473 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:41.847285 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:42.024243 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 08:31:42.049837 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:42.089609 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:42.090950 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:42.091953 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:42.139400 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:42.219827 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 08:31:42.240177 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:31:42.259634 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 08:31:42.262086 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:42.357254 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:42.375413 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:31:42.384677 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:42.385118 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 08:31:42.406100 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:42.434588 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:42.621950 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c4862ea2f67c43a28689b464d8ec96e6: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:42.704349 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:42.785939 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:42.800532 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
0x00007ff51abddd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
I20240502 08:31:42.926787 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:42.943464 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Id Target Id Frame
* 1 Thread 0x7ff50fe37d80 (LWP 8467) "rebalancer_tool" 0x00007ff51abddd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7ff50b9bd700 (LWP 8470) "rebalancer_tool" 0x00007ff51abe732a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7ff5099c4700 (LWP 8471) "kernel-watcher-" 0x00007ff51abe2fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
I20240502 08:31:42.986585 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:42.996882 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:43.019740 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
4 Thread 0x7ff5081c1700 (LWP 21373) "rpc reactor-213" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7ff5071bf700 (LWP 21374) "rpc reactor-213" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7ff5079c0700 (LWP 21375) "rpc reactor-213" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
W20240502 08:31:43.042564 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
7 Thread 0x7ff5089c2700 (LWP 21376) "rpc reactor-213" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7ff5091c3700 (LWP 21377) "rpc reactor-213" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7ff505f7e700 (LWP 22512) "rpc reactor-225" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7ff50577d700 (LWP 22513) "rpc reactor-225" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7ff50479b700 (LWP 22514) "rpc reactor-225" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7ff503f64700 (LWP 22515) "rpc reactor-225" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7ff502f62700 (LWP 23920) "rebalancer_tool" 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7ff502f62700 (LWP 23920)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x6190019fb980, timeout=59.743000000000002) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
W20240502 08:31:43.242941 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:43.250808 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:43.264252 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:43.270619 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:43.286113 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:43.287709 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
#2 0x00007ff51265049e in ev_run (loop=0x6190019fb980, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
W20240502 08:31:43.389496 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=<optimized out>, flags=846848) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff514721c1b in kudu::(anonymous namespace)::ReadFdsFully (progname=..., fds=..., out=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:240
W20240502 08:31:43.462469 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:43.533823 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:43.557590 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:31:43.577936 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:43.602768 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:43.620378 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:43.641868 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:43.668258 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:43.674222 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:43.678717 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
#5 0x00007ff514720d4d in kudu::Subprocess::Call (argv=std::vector of length 12, capacity 16 = {...}, stdin_in=..., stdout_out=<optimized out>, stderr_out=<optimized out>, env_vars=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:773
W20240502 08:31:43.794366 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 08:31:43.906517 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:43.942894 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:43.947090 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:43.984292 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 08:31:43.999102 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:44.027688 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.052038 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.053182 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.061059 21395 catalog_manager.cc:4573] TS 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b88c4a1a36424086aad89b7585c17960: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
W20240502 08:31:44.142671 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 08:31:44.173770 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.270407 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:44.327080 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 08:31:44.410581 21747 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 08:31:44.476507 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:44.502781 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:44.524281 21617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 08:31:44.525497 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:44.535930 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
#6 0x00007ff52a231212 in kudu::tools::RunKuduTool (args=..., out=<optimized out>, err=<optimized out>, in=..., env_vars=std::map with 0 elements) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_test_util.cc:87
#7 0x00000000006278db in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_22::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1255
#8 0x00000000006275bc in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1274
#9 0x000000000062749e in _ZSt13__invoke_implIvZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x000000000062741e in _ZSt8__invokeIZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00000000006273e3 in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::_M_invoke<0ul> (this=0x60300040af58) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x00000000006273ae in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::operator()() (this=0x2f, this@entry=0x60300040af58) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x0000000000627299 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007ff510f796df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7ff503f64700 (LWP 22515)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619001151780, timeout=0.099722762388864794) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619001151780, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x614000079aa0, flags=772480, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000079a98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
W20240502 08:31:44.646085 21486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:31:44.653481 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:44.698833 21485 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 08:31:44.786952 21487 consensus_peers.cc:485] T 12c17796e5d04014aafdac46984f4ba9 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.805764 21618 consensus_peers.cc:485] T 1648586b889b4a1e9ac34481cec6257a P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:31:44.819713 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:31:44.830783 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:44.853106 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:44.865661 21485 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 08:31:44.925204 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:44.965595 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:44.985399 21748 consensus_peers.cc:485] T ebf79c69fd614284ba45fd052cac7ebf P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:45.062044 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:45.174779 21486 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 08:31:45.189643 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:45.207382 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
I20240502 08:31:45.233167 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:45.247630 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:45.260303 20778 raft_consensus.cc:482] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20b637fbec5444dd91b3af9cdda482f5)
I20240502 08:31:45.260594 20778 raft_consensus.cc:504] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cfff665685a14d8796aece26addab59c" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 46845 } } peers { permanent_uuid: "20b637fbec5444dd91b3af9cdda482f5" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45373 } } peers { permanent_uuid: "95f32202b9db457fb09872f6f51ea8ba" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40881 } }
I20240502 08:31:45.261363 20778 leader_election.cc:288] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373), 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881)
W20240502 08:31:45.266111 21487 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
W20240502 08:31:45.288461 21485 leader_election.cc:334] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111)
I20240502 08:31:45.288829 21485 leader_election.cc:302] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfff665685a14d8796aece26addab59c; no voters: 20b637fbec5444dd91b3af9cdda482f5, 95f32202b9db457fb09872f6f51ea8ba
I20240502 08:31:45.295413 20778 raft_consensus.cc:2726] T 203e7914da3f4494b239c6603513caee P cfff665685a14d8796aece26addab59c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:31:45.336900 21616 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000243198) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20240502 08:31:45.428010 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111) [suppressed 29 similar messages]
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7ff50479b700 (LWP 22514)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619001152b80, timeout=0.099384617388750485) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619001152b80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x6140001204a0, flags=774272, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000120498) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000243058) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7ff50577d700 (LWP 22513)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619001153080, timeout=0.099398075389217411) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
W20240502 08:31:45.470702 21618 consensus_peers.cc:485] T d96a7174eb5442f2b0cc0cafc3cc4c80 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
#2 0x00007ff51265049e in ev_run (loop=0x619001153080, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x614000112aa0, flags=228608, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000112a98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000242f18) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20240502 08:31:45.481570 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7ff505f7e700 (LWP 22512)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619001aa1380, timeout=0.099526763388894324) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619001aa1380, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x614000112ca0, flags=230400, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000112c98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000242dd8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
I20240502 08:31:45.508796 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Thread 8 (Thread 0x7ff5091c3700 (LWP 21377)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
I20240502 08:31:45.519428 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619000126180, timeout=0.099682367384502868) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619000126180, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x614000140ca0, flags=231296, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000140c98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000278d98) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7ff5089c2700 (LWP 21376)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x61900012b180, timeout=0.099016292384931148) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x61900012b180, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x6140001600a0, flags=406016, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000160098) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
I20240502 08:31:45.550351 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000278758) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
I20240502 08:31:45.561259 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7ff5079c0700 (LWP 21375)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619000081b80, timeout=0.098984174384895596) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619000081b80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x6140001602a0, flags=406912, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000160298) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000278618) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7ff5071bf700 (LWP 21374)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619000089380, timeout=0.098980107384704752) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619000089380, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x6140001604a0, flags=407808, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000160498) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x6110002784d8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7ff5081c1700 (LWP 21373)):
#0 0x00007ff5109d2a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff51264cde0 in epoll_poll (loop=0x619000088e80, timeout=0.099304919384394452) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007ff51265049e in ev_run (loop=0x619000088e80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007ff516a5e7c1 in ev::loop_ref::run (this=this@entry=0x6140001500a0, flags=408704, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007ff516a56c16 in kudu::rpc::ReactorThread::RunThread (this=0x614000150098) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007ff516a5bd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007ff516a5bbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000278398) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7ff5099c4700 (LWP 8471)):
#0 0x00007ff51abe2fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff5144a13ae in kudu::ConditionVariable::WaitUntil (this=0x612000021b98, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/condition_variable.cc:87
#2 0x00000000006c4dac in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:87
#3 0x00000000006c4c3a in kudu::CountDownLatch::WaitFor (this=0x612000021bc0, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:97
#4 0x00007ff5145d848f in kudu::KernelStackWatchdog::RunThread (this=0x612000021ac0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007ff5145d945c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007ff5145d92d9 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007ff528e6f98d in std::function<void ()>::operator()() const (this=0x611000057d98) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007ff51473301e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
W20240502 08:31:45.767149 21879 consensus_peers.cc:485] T 80b3ba1d834a4314b63f336675845441 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:45.935317 21618 consensus_peers.cc:485] T d4d933eacd944d84b39f0cb9c45641f8 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
Thread 2 (Thread 0x7ff50b9bd700 (LWP 8470)):
#0 0x00007ff51abe732a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000053a46f in __interceptor_waitpid (pid=<optimized out>, status=0x7ff50b9bb6b0, options=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007ff51471d5b6 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:817
W20240502 08:31:45.986313 21879 consensus_peers.cc:485] T 07f5c2dfcd974ee2b8720e9eb052c3f7 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:45.989607 21487 consensus_peers.cc:485] T 59f5d97e6e0a4755be41699246907e55 P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
#3 0x00007ff51471d2e5 in kudu::Subprocess::Wait (this=0x7ff50b9bb6b0, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:544
#4 0x00007ff5145b2638 in kudu::PstackWatcher::RunStackDump (argv=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:258
#5 0x00007ff5145b1d14 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:218
#6 0x00007ff5145b11a9 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:169
#7 0x00007ff5145af537 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:162
#8 0x00007ff528d2263b in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:60
#9 0x00007ff528d224d3 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x00007ff528d22453 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00007ff528d22429 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x60200000d298) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x00007ff528d223e9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x60200000d298) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x00007ff528d222eb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007ff510f796df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
I20240502 08:31:46.049441 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
#15 0x00007ff51abdc6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20240502 08:31:46.057267 21618 consensus_peers.cc:485] T 3898bfdb9c3d4a05af78f79fd2de61e2 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
#16 0x00007ff5109d271f in clone () from /lib/x86_64-linux-gnu/libc.so.6
I20240502 08:31:46.061158 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
Thread 1 (Thread 0x7ff50fe37d80 (LWP 8467)):
#0 0x00007ff51abddd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff510f79933 in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00000000005fbf89 in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody (this=0x61900077f680) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1281
W20240502 08:31:46.087433 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:46.097285 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:31:46.101272 21748 consensus_peers.cc:485] T 852613f14b324b38b5eecc29d4c602cd P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:46.120092 21618 consensus_peers.cc:485] T efec2cc693bd4dd992d0bec22fca5419 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:31:46.127864 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:31:46.340764 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:31:46.355803 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:31:46.389884 21487 consensus_peers.cc:485] T e0241228bd204ca2a8ad2c57b6eeb543 P cfff665685a14d8796aece26addab59c -> Peer fd5e97ae7e344e4687b15ca0cde8d0ba (127.8.68.199:33397): Couldn't send request to peer fd5e97ae7e344e4687b15ca0cde8d0ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.199:33397: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 08:31:46.401690 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:31:46.433012 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:31:46.435825 21617 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 08:31:46.473932 21879 consensus_peers.cc:485] T 76bb1e3e2b9d4e31bf93ac1ce08a7833 P 0f655ab8f20d4ba482c3173f9594b98b -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:46.504926 21746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:31:46.506208 21746 consensus_peers.cc:485] T dc49c050280346db88d1323e30513c80 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 08:31:46.518568 21487 consensus_peers.cc:485] T d62f6eec9ee04f93a3738a83f790292c P cfff665685a14d8796aece26addab59c -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:46.525512 21616 consensus_peers.cc:485] T 182a6f2df91242cfa10f05ef788a19cb P 60677121510e47a5b51d1eb22f9aa238 -> Peer 95f32202b9db457fb09872f6f51ea8ba (127.8.68.200:40881): Couldn't send request to peer 95f32202b9db457fb09872f6f51ea8ba. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:40881: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 08:31:46.594233 21617 consensus_peers.cc:485] T a20b2fd9f1ce488ba7793ed7ac268c16 P 60677121510e47a5b51d1eb22f9aa238 -> Peer 4b9bfd2541144dad86507451343640d3 (127.8.68.198:41299): Couldn't send request to peer 4b9bfd2541144dad86507451343640d3. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:41299: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
#3 0x00007ff5287d70ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7ff5287d882b "the test body", method=<optimized out>, object=0x61900077f680) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#4 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x61900077f680, method=<optimized out>, location=location@entry=0x7ff5287d882b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#5 0x00007ff5287cbbdb in testing::Test::Run (this=0x61900077f680) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
W20240502 08:31:46.608974 21748 consensus_peers.cc:485] T 599d5781f1e24ae9ad425a34d88c6b68 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
#6 0x00007ff5287cbd9d in testing::TestInfo::Run (this=0x612000005a40) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#7 0x00007ff5287cc377 in testing::TestSuite::Run (this=0x612000005bc0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#8 0x00007ff5287cc77c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#9 0x00007ff5287d760d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7ff5287da3e8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x616000006080) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#10 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000006080, method=<optimized out>, location=location@entry=0x7ff5287da3e8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#11 0x00007ff5287cbe63 in testing::UnitTest::Run (this=0x7ff5289eb1e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#12 0x00007ff528d233c2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#13 0x00007ff528d218d9 in main (argc=<optimized out>, argv=0x7ffebfd75688) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109
#14 0x00007ff5108d2bf7 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x00000000005023da in _start ()
I20240502 08:31:46.683537 21923 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:31:46.699793 21748 consensus_peers.cc:462] T 2bfc3eaed583458b8381d4e7b4d36aef P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111)
I20240502 08:31:46.709043 21791 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:31:46.720008 21662 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
************************* END STACKS ***************************
F20240502 08:31:46.757371 8470 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1714638706 (unix time) try "date -d @1714638706" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002113) received by PID 8467 (TID 0x7ff50b9bd700) from PID 8467; stack trace: ***
@ 0x7ff51abe7980 (unknown) at ??:0
I20240502 08:31:46.768473 21530 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7ff5108effb7 gsignal at ??:0
@ 0x7ff5108f1921 abort at ??:0
@ 0x7ff513109dcd google::LogMessage::Fail() at ??:0
@ 0x7ff51310db93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff5131097cc google::LogMessage::Flush() at ??:0
@ 0x7ff51310af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff528d227d9 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at ??:0
@ 0x7ff528d224d3 std::__invoke_impl<>() at ??:0
@ 0x7ff528d22453 std::__invoke<>() at ??:0
@ 0x7ff528d22429 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x7ff528d223e9 std::thread::_Invoker<>::operator()() at ??:0
@ 0x7ff528d222eb std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7ff510f796df (unknown) at ??:0
@ 0x7ff51abdc6db start_thread at ??:0
@ 0x7ff5109d271f clone at ??:0
W20240502 08:31:46.885589 21748 consensus_peers.cc:485] T d8042f6ab6404c5a9f8cce0b8f971542 P 13e764b27ed449c09aee719cdb2597f8 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:31:47.022536 21618 consensus_peers.cc:485] T 8a45b1da348e4bb89f021df103a0603f P 60677121510e47a5b51d1eb22f9aa238 -> Peer 20b637fbec5444dd91b3af9cdda482f5 (127.8.68.197:45373): Couldn't send request to peer 20b637fbec5444dd91b3af9cdda482f5. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.197:45373: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.